cd /news/large-language-models/building-my-own-way-to-compare-llm-b… · home › topics › large-language-models › article
[ARTICLE · art-151004] src=philippdubach.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Building my own way to compare LLM benchmarks

Philipp Dubach built Model Atlas, a tool that compares LLM benchmarks by letting users weight 12 capabilities and score 37 evaluation profiles, with a snapshot dated 15 October 2026 covering 134 benchmark families, 730 observations and 139 configurations. The index draws on Artificial Analysis, Arena Creative Writing, MathArena, JobBench and MCP-Atlas, and keeps 366 release results separate from the combined score because a benchmark's appearance in a release table does not establish that its scores can be combined with another source's results. Model Atlas converts published scores to percentile ranks within each profile's fixed group of tested configurations, so filtering to open weights does not recalculate ranks against a smaller group.

by read2 min views28 publishedOct 13, 2026

The model I would choose for coding may differ from the one I would choose for writing. I built Model Atlas with Codex to compare LLM benchmarks. It started as a Codex artifact. Model Atlas lets me set weights for those capabilities and compare published results for specific model configurations. Something I couldn’t find anywhere else so far.

The full Model Atlas index includes the benchmark catalog, published scores, inference settings, and coverage controls. The snapshot dated 15 October 2026 has 134 benchmark families across 12 capabilities. The index scores 37 evaluation profiles with enough comparable results. They contain 730 observations across 139 configurations.

I set the capability weights to match the work I have in mind. The scored profiles draw on Artificial Analysis, Arena Creative Writing, MathArena, JobBench and MCP-Atlas. For example, JobBench tests professional work. MCP-Atlas tests agents using tools through the Model Context Protocol. In the benchmark catalog, Compare selects one evaluation profile and requires complete coverage. This shows the models tested together under that profile. Use in index adds the profile to a combined score. The 15 profiles added in the latest review start unselected, so the wider catalog does not silently change the default comparison.

By default, each profile converts published scores to percentile ranks within its fixed group of tested configurations. Tied results share the middle rank. Percentiles retain the order of results but discard the size of the gaps between them. Filtering to open weights does not recalculate those ranks against a smaller group.

The index averages the available profile scores within each capability, with equal weight for each profile. My capability weights then determine the combined score. Missing results reduce coverage, which accounts for the selected profiles and their capability weights. At a lower coverage threshold, a model can rank using only part of the selected evidence. That can favour models whose reported results cover their strengths.

The catalog reviews benchmarks from 14 release pages and model cards from Google, Meta, OpenAI, Anthropic, DeepSeek and NVIDIA. It records test methods and limits, including benchmarks that cannot yet support a comparison. Another 366 release results remain separate from the combined score. A benchmark’s inclusion in a release table alone does not establish that its scores can be combined with another source’s results. For example, Meta reports Muse Spark Open at 61.8 on JobBench and 89.2 on MCP-Atlas. The snapshot’s benchmark-owner leaderboards do not list Spark Open. Its JobBench result therefore remains a separate release record. Its MCP-Atlas result enters a separate profile for Meta’s six-model comparison, where Spark Open ranks first. That result does not fill a gap in its Artificial Analysis configuration or determine its rank across other capabilities.

── more in #large-language-models 4 stories · sorted by recency
── more on @model atlas 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/building-my-own-way-…] indexed:0 read:2min 2026-10-13 · —