{"slug": "how-far-do-open-weights-trail-the-frontier", "title": "How far do open weights trail the frontier?", "summary": "A new analysis using Epoch's ECI metric shows that open-weight AI models continue to trail closed models on the frontier, with the gap persisting over time. The analysis, based on item response theory, provides a more nuanced view than the AA Index, revealing that open-weight models have not yet matched the performance of leading closed models like those from OpenAI and Anthropic.", "body_md": "I saw [this Twitter post](https://x.com/yaroslavvb/status/2067367657272422584) today and really liked the idea. But I think the [AA Index](https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-1) is a rather crude way and much prefer [ECI from Epoch](https://epoch.ai/eci), which uses [IRT](https://en.wikipedia.org/wiki/Item_response_theory). The resulting graph does meaningfully diverge from the Twitter post (which seems to weirdly collapse at the end, maybe because of no logistic assumptions being taken into consideration):\n\n**[see linkpost to actually interact with graphs, like seeing what model is what, etc]**\n\nFor context, the two raw frontiers - the running best ECI over time for open-weight vs closed models:\n\nSadly, GLM-5.2 has not been scored yet, but I'll update the website when it is.\n\nYou can also generalize to other criteria (though this is probably the most interesting one). One such example would be the OpenAI vs Anthropic rivalry:", "url": "https://wpnews.pro/news/how-far-do-open-weights-trail-the-frontier", "canonical_source": "https://www.lesswrong.com/posts/DbLnB7eo9wBDDnSEq/how-far-do-open-weights-trail-the-frontier", "published_at": "2026-06-18 11:01:39+00:00", "updated_at": "2026-06-18 11:24:29.316862+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-ethics"], "entities": ["Epoch", "Artificial Analysis", "OpenAI", "Anthropic", "GLM-5.2"], "alternates": {"html": "https://wpnews.pro/news/how-far-do-open-weights-trail-the-frontier", "markdown": "https://wpnews.pro/news/how-far-do-open-weights-trail-the-frontier.md", "text": "https://wpnews.pro/news/how-far-do-open-weights-trail-the-frontier.txt", "jsonld": "https://wpnews.pro/news/how-far-do-open-weights-trail-the-frontier.jsonld"}}