Are We Benchmarking the Wrong Thing? A Humor Leaderboard for LLMs

wpnews.pro

Hmm. When it comes to scoring, there probably isn’t such a thing as the wrong target. Personally, I think evaluation becomes more valuable as the variety of scores increases. At the same time, my intuition is that underused metrics are often overlooked not only because people fail to notice them, but because they are difficult to measure in practice—which probably makes operating a leaderboard like this rather unusual:

Yes, I think this is worth measuring

My direct answer is yes.

Based on the earlier description of the project, the current task appears to be approximately: Given a question and a fixed set of candidate answers, which LLM most reliably predicts the answer that a sampled group of humans will judge as funniest?

That is not the whole of humor, but it is still a meaningful capability. A model that can reliably predict human humor choices may be useful for response reranking, writing assistance, content selection, advertising, roleplay, conversational systems, and other applications where estimating audience reaction matters.

I would therefore frame the current score as something close to:

human humor preference prediction, or population-level humor-choice alignment.

That wording avoids requiring the benchmark to prove that a model has the same internal cognitive process as a human. For practical purposes, reproducible behavior on unseen examples can be valuable even when the internal mechanism is unknown.

The most useful immediate distinction may be:

In other words, the leaderboard does not need a complete theory of humor before it can be useful. It does need enough methodological transparency for readers to know what the number means.

#

What the present score can and cannot establish

A defensible claim would be something like:

Under this item set, voter population, model prompt, inference configuration, and aggregation method, model X agreed with the human target more often than model Y.

That would not automatically establish that model X:

generates better jokes,
makes people laugh more strongly,
understands humor through a human-like internal process,
adapts better to one particular person,
creates a more enjoyable long conversation,
performs equally across languages and cultures,
or is universally “funnier.”

Those are adjacent capabilities, but they are not interchangeable.

The current design may nevertheless capture something useful: the ability to select between competing humorous possibilities in a way that resembles aggregated human judgment.

That is already distinct from ordinary factual accuracy, coding, mathematics, or instruction-following benchmarks.

For practical interpretation, I would reserve a stronger phrase such as “humor understanding” for an explicitly behavioral meaning: The model predicts or reproduces human humor judgments on unseen items with useful accuracy and robustness.

Even then, the score should be tested against simple alternative explanations. A model could be consistent because it always chooses:

the first answer,
the longest answer,
the crudest answer,
the answer containing a particular topic,
or the answer style most reinforced by its post-training.

Repeatability alone is therefore insufficient. Useful evidence would combine:

repeatability,

- held-out performance,
- answer-order robustness,

prompt robustness,
and resistance to simple heuristic baselines.

The incompleteness of the score is not, by itself, a reason to reject it

Every benchmark score is a compression.

Reasoning, coding, safety, instruction following, creative writing, and humor are all broader than the scalar values placed on leaderboards. A score necessarily discards context and dimensions that its designers did not operationalize.

I see two complementary ways to improve evaluation:

Make each individual score more trustworthy.

Better controls
Better sampling
Repeated runs
Confidence intervals
Clearer documentation
Better handling of disagreement

Increase the number of genuinely different evaluation dimensions.

Humor preference prediction
Humor explanation
Creative generation
Personal adaptation
Conversational enjoyment
Multilingual and cultural performance
Appropriateness and safety

This Space contributes mainly to the second direction. It introduces a dimension that is usually absent from general model leaderboards.

That seems valuable even if the resulting score remains narrow.

The larger evaluation problem may therefore be less “we chose the one wrong metric” and more “we rely on too few metrics.”

#

Why several scores should not immediately be collapsed back into one number

Suppose the project eventually produced the following measurements:

It would be tempting to assign weights and produce one “Overall Humor Score.”

That would be convenient, but the weights would encode a use case and a value judgment.

A roleplay model, a public content-ranking system, a writing assistant, a companion chatbot, and a general-purpose assistant do not necessarily require the same combination of abilities.

I would therefore distinguish:

A multidimensional profile that supports an overall human judgment

from:

A universal weighted scalar claiming to rank every model for every purpose

The first is easier to defend and more reusable.

The current methodology is the part that most needs precision

Several implementation choices would materially change what the leaderboard measures.

The most immediately useful improvement may therefore be an expanded README or a short METHODOLOGY.md

, rather than a more complicated scoring model.

At minimum, readers should be able to identify:

where the questions and candidate answers came from,
how human votes were collected,
the exact question shown to human voters,
the exact instruction given to the models,
how the human target was aggregated,
how the score was calculated,
how answer order was handled,
how many times each model was evaluated,
and which exact model versions and inference settings were used.

This does not require a full academic paper. Hugging Face’s conventions for Model Cards, Dataset Cards, and Evaluation Results already provide useful documentation patterns that could be adapted to a Space.

A practical minimum

#

Human and model wording can define different benchmarks

These human questions are not equivalent:

“Which answer is funniest to you?”
“Which answer would most people find funniest?”
“Which answer made you laugh the most?”
“Which answer is the cleverest?”
“Which is the best response?”

Likewise, these model instructions are not equivalent:

“Choose the answer you find funniest.”
“Predict which answer humans selected.”
“Predict which answer the average voter would select.”
“Estimate the distribution of human votes.”

The first asks for something resembling a model-side judgment.

The others ask for social prediction.

Both are interesting, but they define different targets. Publishing the exact wording would prevent readers from silently substituting one interpretation for another.

The same applies to human aggregation.

If an item receives: then a model selecting B is different from a model missing an item where:

A majority label treats both cases identically. An average-agreement score or distributional score would not.

None of these methods is universally correct; the important point is to expose which one is being used.

A small set of controls would make the ranking much easier to trust

The visible ranking layer deserves relatively high rigor because readers naturally interpret ranks as comparative facts.

The following checks would eliminate many alternative explanations without requiring a major redesign:

Rotate or randomize candidate order.
Repeat a subset of model/item evaluations.
Test one semantically equivalent prompt variant.
Compare against random and simple heuristic baselines.
Preserve a held-out or newly collected item set.
Report performance separately for high- and low-consensus human items.
Display sample counts and uncertainty.
Avoid presenting statistically indistinguishable models as decisively ordered.

By contrast, exploratory analyses of model families, humor categories, or unusual failure cases can begin with lighter methodology—as long as they are described as hypothesis generation rather than causal conclusions.

#

The closest precedent shows why these controls matter

The most direct neighboring work I found is Cards Against LLMs. Five frontier models were given the same ten-card choices as human Cards Against Humanity players across 9,894 rounds. All models performed above the random baseline, but agreement with humans remained modest. The models agreed with one another more strongly than they agreed with humans.

The authors also found systematic position and content preferences that explained part of the inter-model agreement.

This does not prove that the current Space has the same biases. The datasets, prompts, candidate structures, and aggregation methods may differ.

It does establish that, for this class of task:

answer order is worth testing,
repeated-run consistency is worth measuring,
model–model agreement is informative,
and a high score can contain both genuine predictive signal and structural shortcuts.

Another close case is Cards Against AI, which used 300,000 online Cards Against Humanity games to predict the human winner. Its models performed above random without user information, but analysis suggested heavy reliance on punchline-side properties and preferences for short, crude, juvenile answers.

Again, the point is not that this Space necessarily behaves the same way. The point is that successful prediction and complete understanding are not equivalent, and targeted controls can show what information the predictor is using.

The leaderboard could become a useful probe of model families

Even if the construct remains somewhat broad, applying the same task to many models could reveal useful behavioral regularities.

Examples include:

whether performance generally increases with model scale,
whether that effect saturates,
whether different model families form different preference profiles,
whether reasoning tuning transfers to humor judgment,
whether coding specialization reduces, preserves, or improves it,
whether roleplay or creative-writing tuning helps,
and whether multilingual training changes cross-language performance.

The first result would be a correlation or pattern, not a causal explanation. That is still useful because it identifies where a more controlled comparison is worth running.

A leaderboard can therefore serve two roles:

Ranking: comparison under one defined measurement. Probe: discovery of patterns that motivate follow-up experiments.

#

A useful order for model comparisons

The cleanest available comparisons would be:

Variants of the same base model

base vs instruct
instruct vs reasoning
general vs coder
general vs roleplay or writing tune

These do not perfectly isolate one factor, but they are often closer comparisons than unrelated models.

Sizes within the same family and generation

For example: This can reveal whether there is a scale trend and whether it is monotonic.

Similar size and generation across families

For example:

Qwen
Gemma
Llama
Mistral-family models Such comparisons remain confounded by data, architecture, tokenizer, and post-training, so they are best interpreted as family profiles rather than isolated architecture effects.

Specialization differences

general vs coding
general vs mathematics
general vs reasoning
general vs creative writing
general vs roleplay

Interesting outcomes could occur in either direction.

A coding or reasoning model might lose some sensitivity to social preference. It might also gain transferable structural reasoning that helps identify wordplay, reversal, or incongruity.

HumorBench reports that STEM-oriented reasoning improvements can transfer to humor explanation, and that increasing thinking budget has different effects across models. That is a different task from this leaderboard, so it does not predict the result here. It does make the comparison worth running.

Inference configuration

reasoning enabled or disabled
different thinking budgets
deterministic vs sampled runs
repeated evaluations

A model that improves with more reasoning on explanation tasks might become worse at predicting spontaneous human preference, or vice versa.

Language specialization

multilingual models
language-focused models
translated items
native-cultural items This would help separate language processing from cultural prediction, although it would not remove all confounding factors.

The safe form of reporting would be:

“In this sample and configuration, coder variants scored lower.”

The stronger claim:

“Coding training destroys humor ability.”

would require controlled experiments beyond a leaderboard correlation.

“Funny” is not one scalar human response

A one-click selection is operationally useful, but the human reaction compressed into that click may take several forms:

a small chuckle,
a grin,
a large laugh,
delayed amusement,
admiration for a clever analogy,
recognition of an elegant construction,
appreciation of insight or satire,
surprise,
discomfort mixed with amusement,
or “I do not personally like this, but I expect many other people will.”

These are not merely different intensities of one identical response.

A useful conceptual map is:

The main leaderboard does not need to encode every dimension. This map instead clarifies what the main score necessarily compresses.

#

Related projects that divide humor into different tasks or dimensions

Oogiri-Master studies Japanese Oogiri with large candidate sets and many independent human judges. It also investigates features such as response length, ambiguity, and incongruity resolution.

Assessing the Capabilities of LLMs in Humor evaluates Oogiri responses across Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness. It reports that humans and LLMs can weight these qualities differently.

HumorBench evaluates explanation of the conceptual and cultural components necessary to understand sophisticated cartoon humor.

The New Yorker Caption Contest work separates matching, ranking, and explanation rather than treating humor as one undifferentiated capability. The paper Do Androids Laugh at Electric Sheep? and its public corpus and code are useful references.

A later dataset, Humor in AI, released more than 250 million human ratings on over 2.2 million New Yorker captions. This shows that direct human evaluation of generated humor is possible, but it also illustrates the amount of preference data that may be required.

These approaches are complementary:

selection measures preference prediction,
explanation measures mechanism recognition,
generation measures creative production,
and multidimensional annotation shows why a single funniness score was assigned.

Population-level humor and interpersonal humor should remain separate targets

A system that predicts what a broad voter population will prefer is solving a public-audience problem.

A system trying to make one particular person laugh during an ongoing conversation must also account for:

the person’s knowledge,
relationship with the speaker,
shared history,
current mood,
social boundaries,
earlier successful and failed jokes,
timing,
and how to recover when a joke lands badly.

These are different skills.

“Something mildly funny to many people” and “something extremely funny to this one person” may require opposite choices.

I would therefore distinguish:

Population Humor Alignment Interactional Humor Adaptation

The current Space appears much closer to the first. That is a scope boundary, not necessarily a defect.

#

Item-level prediction is also different from conversation-level enjoyment

There is another granularity distinction:

Item-level preference

Which isolated answer is funniest?

Turn-level enjoyment

Did this response improve the interaction?

Conversation-level enjoyment

Was the exchange entertaining overall?

Long-term interpersonal fit

Did the model learn the user’s preferences and stop repeating failed humor?

A model can perform well at one level and poorly at another.

The paper Role of Reasoning in LLM Enjoyment Detection examines enjoyment at different conversational levels and finds that results depend on which level and human signal are used. It is not the same task as this Space, but it is a useful reason not to generalize an isolated answer-selection score into complete conversational enjoyment.

Roleplay and personalized-preference projects may eventually offer components for the interpersonal side.

For example, the RP-Bench repository combines multidimensional evaluation with a community voting arena and explicitly compares LLM-as-judge signals with user preference. The Creative Writing Benchmark similarly treats subjective creative quality as a combination of comparative ranking and more specific criteria rather than one vague absolute judgment.

Human disagreement may be part of the signal

For a subjective task, the shape of the vote distribution may be nearly as informative as the winner. These two results produce the same majority label:

- A: 98%, B: 2%
- A: 51%, B: 49%

But they represent very different human judgments.

On the second item, a model selecting B agrees with almost half the voters. Treating it as simply wrong discards meaningful information.

Possible diagnostic displays include:

winner share,
number of votes,
human agreement or entropy,
accuracy on high-consensus items,
accuracy on divided items,
and distance from the complete human vote distribution.

The general argument in We Need to Consider Disagreement in Evaluation is relevant here: subjective annotation disagreement should not always be collapsed into one gold label without examining what is being lost.

Not every disagreement is meaningful diversity. Some comes from misunderstanding, ambiguous items, inattention, spam, or poor instructions. Preserving distributions and applying quality control are complementary rather than competing goals.

#

Optional deeper annotation can preserve information without breaking the core interaction

Requiring every voter to supply:

reaction type,
intensity,
cultural background,
reference familiarity,
appropriateness,
and a written explanation

would probably make the Space much harder to operate.

A useful analogy is the difference between a low-friction star rating and a written review:

the first gathers a broad but shallow signal;
the second contains richer information but comes from a much more self-selected subset.

The main event can remain:

select one answer → submit vote Deeper information could be collected through one of several optional paths.

Option A: Expand on demand

Add an optional note

Option B: Ask one randomly selected follow-up

Examples:

How strong was your reaction?
Had you seen a similar joke before?
Did you understand the reference?
Was it funny but also uncomfortable?
Did you personally enjoy it, or mainly expect others to?

Option C: Provide a separate deep-annotation mode

A smaller, explicitly self-selected group can contribute detailed labels without changing the normal voter flow.

For all three options:

- report optional-response coverage,

do not treat missing responses as negative answers,
do not assume optional respondents represent all voters,
keep the original vote compatible with historical data,
and initially keep optional annotations separate from the main score.

If the UI or question wording changes, the version should also be recorded so that a measurement change is not mistaken for a model or population change. Multilingual evaluation could expose especially interesting differences

A multilingual extension should separate at least three problems:

Language processing

Can the model understand the item?

Translation robustness

Does the humorous structure survive translation?

Cultural alignment

Can the model predict the preferences of people in that cultural context?

A translated English joke and a joke created naturally in Japanese, Chinese, Arabic, Spanish, or another language community are not equivalent items.

Translation can destroy:

wordplay,
rhythm,
ambiguity,
social register,
politeness effects,
cultural references,
or implied meaning.

A model can also be linguistically fluent while lacking the relevant shared knowledge. Conversely, it may recognize the reference while failing to predict whether the target audience currently finds it funny.

Oogiri-Master and Chumor 2.0 are useful examples because they treat Japanese and Chinese humor as locally situated problems rather than only translated English benchmarks. Chumor also provides a Hugging Face dataset, a leaderboard, and public code.

#

A possible multilingual data layout

Useful item metadata could include:

source language,
native or translated item,
cultural origin,
required background knowledge,
estimated reference familiarity,
collection date,
voter language,
and human agreement.

The evaluation could then expose separate views:

Translation view

The same or closely matched content across languages.

This primarily tests language handling and translation robustness, although perfect equivalence may be impossible for wordplay.

Native-cultural view

Items originally created within each linguistic or cultural community and rated by relevant speakers.

This better captures locally grounded humor, but raw scores should not be treated as automatically comparable across datasets of different difficulty.

Cross-cultural prediction view

Whether a model—or a voter group—can predict the preferences of a different cultural group.

A multilingual model performing well would be an interesting result, but not sufficient proof of general cultural understanding. Model scale, data exposure, fluency, and post-training may all contribute.

That uncertainty does not make the result useless. It identifies the next comparison to run.

Disagreement patterns could help locate possible “fault lines” behind humor

Even without a complete theory of humor, clusters of agreement and disagreement may reveal where models and humans are responding to different cues.

Possible “fault lines” include:

expected continuation vs surprising outcome,
one semantic frame vs another,
literal vs pragmatic meaning,
shared knowledge vs missing reference,
familiar template vs novelty,
harmless play vs norm violation,
public majority vs individual taste,
and text content vs timing or delivery.

I use “fault line” only as an exploratory metaphor. A leaderboard result does not directly reveal internal cognition.

Its value would be identifying where a controlled follow-up experiment is worth building.

#

A path from an interesting leaderboard pattern to a targeted experiment

Suppose one model family performs well on wordplay but poorly on cultural-reference items.

A small follow-up set could alter one feature at a time:

replace an ambiguous word with an unambiguous synonym,
explain the cultural reference,
remove the punchline,
make the ending predictable,
preserve meaning while matching answer length,
weaken or strengthen a norm violation,
or rotate candidate position.

The comparison would then examine:

the human vote shift,

- the model-choice shift,
- and repeated-run stability.

This would not establish a complete theory of humor. It would test whether the proposed feature contributes to the observed choice.

Getting Serious about Humor offers a related approach: models edit humorous text into aligned non-humorous counterparts. Such “funny/unfunny” counterfactual pairs can be more diagnostic than comparing unrelated jokes.

A practical research path could therefore be:

leaderboard item → disagreement cluster → small counterfactual set → targeted test

This allows the Space to remain lightweight while still generating material for stricter research.

A staged roadmap can preserve the Space’s simplicity

The project does not need to adopt every possible extension.

Stage 1 — Make the current score interpretable

Publish the exact human and model prompts.
Describe item and vote sources.
Publish the score formula.
Record exact model versions and evaluation dates.
Rotate or randomize candidate positions.
Show sample counts and uncertainty.

Stage 2 — Make the score diagnostically useful

Preserve the human vote distribution.

- Separate high- and low-consensus items.
- Add model self-consistency.

Add simple heuristic baselines.
Add family, size, generation, and specialization views.
Preserve historical leaderboard versions.

Stage 3 — Add optional depth

Optional reaction type
Optional intensity
Optional familiarity or reference knowledge
Optional personal-funniness vs predicted-popularity distinction
Optional explanation
Explicit annotation coverage

Stage 4 — Add independent research tracks

Humor explanation
Counterfactual understanding
Human-ranked generation
Native multilingual and cultural sets
Individual preference adaptation
Conversation-level enjoyment These are different levels of ambition. The Space can remain valuable at Stage 1 or Stage 2 without becoming a complete humor research platform.

Overall

I would frame this less as:

a definitive ranking of which model truly understands humor

and more as:

an exploratory leaderboard for how different models predict a particular population’s relative humor choices under controlled items.

That leaves room for several legitimate outcomes:

a practical population-preference score,
a new dimension in the broader model-evaluation landscape,
a probe for model-family and specialization differences,
a source of unusual failure cases,
and a starting point for studying where human and model humor judgments diverge.

So yes, I think “making people smile” is worth tracking.

The requirement is not that one score capture every meaning of humor. The requirement is that readers can tell what the score measures, under which conditions, and with what uncertainty.

Once that foundation is clear, the ambiguity of humor becomes not only a limitation, but also part of what makes the Space interesting.

source & further reading

discuss.huggingface.co — original article Rakarrack-0.6.1 port making progress! ( AI assisted ) Cloud Storage Poll Welcome to Haiku basic(Haiku Docs, Haiku slide and Haiku sheets)

Are We Benchmarking the Wrong Thing? A Humor Leaderboard for LLMs

Run your AI side-project on zahid.host