{"slug": "why-mcq-based-evaluation-beats-bleu-for-video-captions", "title": "Why MCQ-based evaluation beats BLEU for video captions", "summary": "A proposed evaluation method for video captioning replaces BLEU and METEOR with Multiple-Choice Question Answering (MCQA), scoring captions by the percentage of video-grounded questions an LLM can answer correctly from the caption text alone. The author reports that captions with low BLEU scores can still hit 90% on the MCQA \"Fidelity Score,\" while noting the approach requires 4,000+ LLM calls for a 1,000-video dataset and can run on GPT-4o or a smaller specialized model to control cost. The method's stated failure point is distractor quality, since visually similar but factually wrong options are needed to prevent the evaluator from guessing.", "body_md": "# Why MCQ-based evaluation beats BLEU for video captions\n\nTraditional metrics like BLEU or METEOR are useless for video captioning because they penalize any word that doesn't match a ground-truth reference, even if the description is accurate. I've found that shifting to Multiple-Choice Question Answering (MCQA) is the only way to actually measure information fidelity. Instead of checking if a caption \"looks like\" a reference, you use an LLM to ask if the caption contains the correct facts about the video, treating the caption as the source of truth.\n\n## How the MCQA evaluation process actually works\n\nThe goal is to see if a caption maximizes coverage of the video's content without hallucinating. I've been testing this by generating a set of questions where only one answer is supported by the video. If the model can answer the question correctly using only the generated caption, the caption is considered high-fidelity.\n\n1. **Question Generation**: Use a VLLM to generate a set of questions based on the video and the ground-truth reference.\n\n2. **Distractor Creation**: Generate three incorrect but plausible options (distractors) for each question.\n\n3. **Caption-based Answering**: Feed the generated caption into a separate LLM. The model must choose the correct answer among the four options based solely on the text of the caption.\n\n4. **Scoring**: The \"Fidelity Score\" is the percentage of questions correctly answered. If the model picks the wrong option, it means the caption either missed a key detail or introduced a hallucination.\n\n## Where this breaks compared to reference matching\n\nWhen I ran this against standard benchmarks, the discrepancy was jarring. A caption could have a very low BLEU score because it used different adjectives or focused on a different part of the scene, yet it would score 90% on the MCQA test because the factual density was high.\n\nThe failure point here is the \"distractor quality.\" If the distractors are too obvious, the LLM might guess the right answer even if the caption is vague. To get a real reading, the distractors need to be visually similar to the actual scene but factually wrong.\n\n## Performance trade-offs and costs\n\nThis method is computationally expensive. While a BLEU score takes milliseconds to calculate, an MCQA evaluation requires multiple LLM passes.\n\n- **Latency** : For a dataset of 1,000 videos, you are looking at 4,000+ LLM calls (1 for question generation and 3+ for distractor validation per video).\n- **Cost** : Using GPT-4o for the evaluation phase can get pricey. I've had better luck using a smaller, specialized model for the answering phase to keep costs down, though you risk \"model bias\" where the evaluator is too lenient on the captioner.\n- **Accuracy** : It eliminates the \"one-to-many\" problem where a video of a dog running in a park could be described as \"a golden retriever in the grass\" or \"a dog sprinting through a field.\" Both are correct, but reference matching would penalize one of them.\n\n[Next DiscoSign handles sign language glosses better than standard LLMs by tracking spatial →](/en/threads/9217/)\n\n[these real-world AI monetization case studies](https://tanyan888.com/), with plenty of directly applicable cases.\n\n## All Replies （3）\n\nI want to try this tonight. You missed the hallucination problem, specifically with tools like CLIPScore or...\n\nFinally! My old boss used BLEU to judge my captions and I almost quit. Does this work with the 0.8 threshold?\n\nCurious if this scales. Does the MCQ approach hold up when you increase the distractor count to 5 or more?", "url": "https://wpnews.pro/news/why-mcq-based-evaluation-beats-bleu-for-video-captions", "canonical_source": "https://promptcube3.com/en/threads/9218/", "published_at": "2026-09-11 16:24:02+00:00", "updated_at": "2026-09-11 17:13:45.127715+00:00", "lang": "en", "topics": ["large-language-models", "computer-vision", "natural-language-processing", "ai-research", "ai-tools"], "entities": ["BLEU", "METEOR", "Multiple-Choice Question Answering", "GPT-4o", "CLIPScore", "DiscoSign"], "alternates": {"html": "https://wpnews.pro/news/why-mcq-based-evaluation-beats-bleu-for-video-captions", "markdown": "https://wpnews.pro/news/why-mcq-based-evaluation-beats-bleu-for-video-captions.md", "text": "https://wpnews.pro/news/why-mcq-based-evaluation-beats-bleu-for-video-captions.txt", "jsonld": "https://wpnews.pro/news/why-mcq-based-evaluation-beats-bleu-for-video-captions.jsonld"}}