{"slug": "openais-chief-scientist-wants-perplexity-to-be-ais-speedometer", "title": "OpenAI’s chief scientist wants perplexity to be AI’s speedometer", "summary": "OpenAI chief scientist Jakub Pachocki published a November 22, 2026 blog post, \"Perplexity and Intelligence,\" proposing log-perplexity (cross-entropy loss) as a shared cross-lab metric for tracking pretraining progress and disclosing OpenAI's internal held-out scores: approximately 0.99 for GPT-3.5, about 0.91 for GPT-4, below 0.8 for newer models, and around 0.76 for a breakthrough math model. Pachocki flagged roughly 0.68 to 0.70 as the range where transformative capabilities are expected to emerge and recommended developers avoid crossing that threshold until adequate monitoring frameworks are in place, building on Anthropic CEO Dario Amodei's September 2026 \"pace the frontier\" plan. The proposal's open question is comparability, since OpenAI's scores were measured on its own held-out dataset and a cross-lab limit would require other developers to measure against the same data or something calibrated to it.", "body_md": "# OpenAI’s chief scientist wants perplexity to be AI’s speedometer\n\nFoxTPNL / Wikimedia Commons (CC BY 4.0)\n\nJakub Pachocki published OpenAI's internal pretraining scores and proposed shared limits before models cross a key threshold\n\n[OpenAI](https://cryptobriefing.com/markets/openai/) chief scientist Jakub Pachocki thinks the AI industry needs a speedometer before it can agree on a speed limit. In a blog post titled “[Perplexity](https://cryptobriefing.com/markets/perplexity/) and Intelligence,” published November 22, 2026, he nominated a candidate: a measurement called log-perplexity.\n\nThe post also does something OpenAI has not done before. It publicly discloses the company’s internal pretraining log-perplexity scores on a held-out dataset, putting real numbers behind a debate that has mostly run on vibes and essays.\n\n## What Pachocki actually proposed\n\nLog-perplexity, also known as cross-entropy loss, is a way of scoring how well a language model predicts text it has not seen before. A model that is constantly caught off guard by the next word scores high. A model that sees it coming scores low.\n\nLower, in this case, means more capable. Pachocki is arguing that this single number can track pretraining progress across labs and serve as a trigger point for safety monitoring.\n\nThe internal figures he shared tell a tidy story of descent:\n\n- **GPT-3.5:** approximately 0.99\n- **GPT-4:** about 0.91\n- **Newer models:** below 0.8\n- **A breakthrough math model:** around 0.76\n\nThose numbers matter because of where Pachocki says the danger zone sits. He flagged the window of roughly 0.68 to 0.70 as the range where transformative capabilities are expected to emerge.\n\nHis recommendation is that developers avoid crossing that threshold until adequate monitoring frameworks are in place. The math model at around 0.76 is not exactly parked far from the line.\n\nHe also points to a potential rapid advancement in pretraining capabilities, with significant improvements possible as early as 2027.\n\n### AI, tech, and the markets they move—in one daily briefing.\n\nDaily. Free. Join 34,000+ readers across crypto, finance, and policy.\n\n## Why a measurement comes before a limit\n\nThe post lands in the middle of a broader push to slow the frontier on purpose. In September 2026, [Anthropic](https://cryptobriefing.com/markets/anthropic/) CEO Dario Amodei published a plan to “pace the frontier,” a framework for keeping AI development from outrunning the safeguards around it.\n\nThat plan was seemingly endorsed by OpenAI CEO Sam Altman and SpaceX CEO Elon Musk.\n\nBut a pacing agreement has an obvious gap. You cannot enforce a speed limit if every driver uses a different speedometer, or refuses to show theirs. Pachocki’s post is an attempt to fill that gap by offering a common unit of measurement that labs could report against.\n\nIt builds directly on Amodei’s framework rather than competing with it. Anthropic supplied the idea of a brake. OpenAI is now proposing the gauge that tells everyone when to press it.\n\n## The case for perplexity, and the catch\n\nPerplexity has some real advantages as a shared yardstick. It is a single number, it is computed during pretraining, and it reflects raw predictive power rather than how a model performs on a particular benchmark that can be gamed or memorized.\n\nThe catch is comparability. Pachocki’s scores were measured on OpenAI’s own held-out dataset. For a cross-lab limit to mean anything, other developers would need to measure against the same data, or something calibrated to it, and agree on how to report results honestly.\n\nThere is also the question of whether a single loss figure maps cleanly onto the capabilities people actually worry about. Pachocki’s proposal effectively bets that it does, at least well enough to draw a line around the 0.68 to 0.70 range.\n\n## What this means for the AI industry\n\nThe most concrete shift here is transparency. OpenAI has historically been guarded about training internals, so publishing its own pretraining scores is a notable change in posture. It also creates a reference point that competitors will now be implicitly measured against, whether or not they choose to publish.\n\nThe things to watch are fairly specific. First, whether Anthropic, and other frontier developers, publish comparable log-perplexity figures. Second, whether anyone proposes a neutral, shared held-out dataset that would make those numbers directly comparable. Third, what “adequate monitoring” is defined to mean, since the whole proposal hinges on that milestone being concrete enough to verify.\n\nFor now, the industry has a proposed metric, a published set of reference scores, and a rough red line. Whether that adds up to an actual speed limit depends on how many drivers agree to install the same gauge.\n\n**Disclosure:** This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our [Editorial Policy](https://cryptobriefing.com/editorial-policy/).", "url": "https://wpnews.pro/news/openais-chief-scientist-wants-perplexity-to-be-ais-speedometer", "canonical_source": "https://cryptobriefing.com/openai-pachocki-perplexity-ai-benchmark/", "published_at": "2026-11-22 21:17:48+00:00", "updated_at": "2026-11-22 22:12:22.602657+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["OpenAI", "Jakub Pachocki", "Anthropic", "Dario Amodei", "Sam Altman", "Elon Musk", "GPT-3.5", "GPT-4"], "also_reported_by": [], "takeaway": null, "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/openais-chief-scientist-wants-perplexity-to-be-ais-speedometer", "markdown": "https://wpnews.pro/news/openais-chief-scientist-wants-perplexity-to-be-ais-speedometer.md", "text": "https://wpnews.pro/news/openais-chief-scientist-wants-perplexity-to-be-ais-speedometer.txt", "jsonld": "https://wpnews.pro/news/openais-chief-scientist-wants-perplexity-to-be-ais-speedometer.jsonld"}}