{"slug": "llm-evaluation-explained-how-a-benchmark-produces-comparable-numbers", "title": "LLM Evaluation Explained: How a Benchmark Produces Comparable Numbers", "summary": "The LFORLA Reverse Engineering benchmark scores LLMs on restoring C source from stripped binaries using identical prompts, a deterministic token-similarity judge, and four per-axis rubrics with a public verbatim answer trail. LFORLA's own submission, GLM 5.2, scored 78.0 overall, while the current leaderboard entry Nemotron 3 Ultra scored 43.48991396255626 under a blended 60 percent cosine and 40 percent LCS metric, with Function Recovery requiring token similarity of at least 0.75.", "body_md": "LLM evaluation only becomes useful when every model faces the same prompts, the same fixed judge, and the same per-axis rubrics, with a public verbatim trail so anyone can re-read what a model actually answered. This article explains that mechanism step by step, then shows the real numbers from the LFORLA Reverse Engineering benchmark, where our own submission GLM 5.2 scored 78.0 overall and the current leaderboard entry Nemotron 3 Ultra scored 43.48991396255626.\n\nComparable numbers do not appear by accident. They are the output of a fixed pipeline. Here is the mechanism, using the Reverse Engineering benchmark as the example.\n\nThe benchmark is called Reverse Engineering (Binary to Source). The task is to restore the C source of a function from its stripped binary. Models can use tools or no-tools mode. Every model receives the same prompt and the same binary. Nothing changes between runs except the model.\n\nScoring is deterministic token-similarity against server-only references. The references are not shown to the model. The judge does not improvise. It compares tokens, not vibes. That is what makes the numbers comparable across models and across time.\n\nA single number hides too much, so the benchmark reports four axes:\n\nEvery submission is kept so anyone can re-read what a model actually answered. If a score looks surprising, you can open the answer and check the tokens yourself. That is the difference between a leaderboard and a black box.\n\n| Model | Provider | Score | \n|---|---|---|\n| Nemotron 3 Ultra (free, via opencode) | deepseek | 43.48991396255626 | \n| GLM 5.2 (our own submission) | lforla | 78.0 | \n\nThe gap between 78.0 and 43.48991396255626 is not a rounding artifact. It reflects how much of the reference source each model actually reconstructed. A score near 78 means the submission covered most of the reference tokens and matched them closely under the blended 60 percent cosine and 40 percent LCS metric. A score near 43 means roughly half the reference was missing or misaligned. Function Recovery is stricter still: it only fires when token similarity reaches 0.75, so a model can look decent on Completeness and still fail the binary gate. In practice, the axes tell you where a model is strong and where it is guessing.\n\nDo not pick a model from one headline number. Pick it from the axis that matches your workload. If you need full source reconstruction, look at Completeness and Function Recovery. If you need close token-level fidelity, look at Code Similarity. If you need a general signal, look at Percent Reversed. Then read the verbatim trail for the cases you care about. A model that scores well on easy binaries and fails on hard ones is not the same as a model that scores consistently.\n\nToken similarity is not semantic correctness. A submission can match tokens and still be wrong in structure. The 60 percent cosine and 40 percent LCS blend is a design choice, not a universal truth. The Reverse Engineering benchmark covers one task family, so results do not automatically transfer to other code or reasoning tasks. The leaderboard shown here is a snapshot, and scores can move as more models are submitted. Our own submission GLM 5.2 is included, so treat that number with the same scrutiny as any other.\n\nLLM evaluation is a mechanism, not a mystery. Same prompts, same fixed judge, per-axis rubrics, and a public verbatim trail are what turn raw answers into comparable numbers. You can inspect the full benchmark and leaderboard at [https://lforla.org](https://lforla.org). If you want to see how your own model scores on Reverse Engineering, submit it and read the trail yourself.", "url": "https://wpnews.pro/news/llm-evaluation-explained-how-a-benchmark-produces-comparable-numbers", "canonical_source": "https://dev.to/resk/llm-evaluation-explained-how-a-benchmark-produces-comparable-numbers-52bc", "published_at": "2026-10-12 09:00:36+00:00", "updated_at": "2026-10-12 09:09:54.251089+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "ai-tools"], "entities": ["LFORLA", "GLM 5.2", "Nemotron 3 Ultra", "deepseek", "opencode"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-evaluation-explained-how-a-benchmark-produces-comparable-numbers", "markdown": "https://wpnews.pro/news/llm-evaluation-explained-how-a-benchmark-produces-comparable-numbers.md", "text": "https://wpnews.pro/news/llm-evaluation-explained-how-a-benchmark-produces-comparable-numbers.txt", "jsonld": "https://wpnews.pro/news/llm-evaluation-explained-how-a-benchmark-produces-comparable-numbers.jsonld"}}