{"slug": "your-ai-can-say-gravity-without-knowing-what-it-means", "title": "Your AI Can Say “Gravity” Without Knowing What It Means", "summary": "A new analysis of vision-language models (VLMs) shows they fail basic physics reasoning, with the average model scoring around 40% on the PhysBench benchmark (ICLR 2025) and the best model, GPT-4o, scoring 49.49%, compared to humans at 95%. Meta's IntPhys 2 benchmark found most state-of-the-art models perform at chance (50%) on tasks infants can solve, and a 2025 failure-mode study of six frontier VLMs concluded they often answer correctly for the wrong reasons, relying on correlation rather than causal physics.", "body_md": "**Hello DataChefs!** 👋 Welcome back.\n\nA few weeks ago I showed a vision-language model a photo of a cup hanging half off a table. It said the cup was about to fall.\n\nCorrect answer. Wrong reasoning.\n\nIt had no idea where the cup’s center of mass was, whether it was full or empty, or whether someone had just bumped the table. It got the right call because “cup near edge” almost always gets captioned as “about to fall” somewhere in its training data.\n\nThat’s not physics. That’s a very good guess wearing physics vocabulary.\n\n⚠️ Fair warning: this article contains benchmark numbers that will make you side-eye every “AI understands the physical world” headline you see for the next six months. Proceed accordingly.\n\nIf you’re new here, this article is part of an ongoing obsession. In [Model Recovery vs Model Learning](https://medium.com/@farhat.shaikh/model-recovery-vs-model-learning-the-great-divide-in-data-science-aa5af7e06721), I argued that a model can behave correctly without knowing why. In [SINDy vs PINNs](https://medium.com/@farhat.shaikh/can-ai-discover-the-laws-of-physics-sindy-vs-pinns-explained-for-data-scientists-33ecb8b589b7), I walked through two ways AI can recover the actual governing equations of a system. And in [What Is a World Model?](https://pub.towardsai.net/what-is-a-world-model-inside-the-ai-idea-behind-2026s-1-billion-bet-f98986c4d779), I traced why the industry is betting a billion dollars on models that predict rather than describe.\n\nToday I’m pointing all of that at Vision-Language Models, because everyone’s suddenly convinced they understand the physical world since they can narrate a video.\n\nMy honest answer: not reliably. And unlike most hot takes on this topic, I brought numbers.\n\nLet’s start with what happens when researchers stop asking VLMs to caption pretty pictures and start testing them on physics.\n\n[PhysBench](https://arxiv.org/abs/2501.16411), an ICLR 2025 benchmark with 10,002 video-image-text questions covering physical properties, relationships, scenes, and dynamics, evaluated **75 VLMs**. The average model scored around 40%. The best model at the time, GPT-4o, managed 49.49%. Humans sit around 95%.\n\nMeta’s [IntPhys 2](https://arxiv.org/abs/2506.09849) is even more brutal. It tests four things infants figure out before they can talk: object permanence, immutability, spatio-temporal continuity, and solidity. Models watch paired videos where one outcome is physically possible and one is impossible, and simply pick which is which.\n\nMost state-of-the-art models score **at chance**. 50%. A literal coin flip. Humans are near-perfect.\n\nSit with that. The model that eloquently explains gravity in a caption cannot reliably tell you that an object shouldn’t pass through a solid wall.\n\nRemember my cup experiment? Turns out researchers ran the rigorous version of it.\n\nA [2025 failure-mode study](https://arxiv.org/abs/2510.06251) tested six frontier VLMs on the Physion, Physion++, and CLEVRER benchmarks, then built subtests to check whether models were actually attending to the right objects and dynamics behind their answers. The finding, in the authors’ own framing: models often achieve benchmark scores **for the wrong reasons**, answering correctly without grounding in perception or physics.\n\nWhy? Because VLMs are trained to predict language from pixels. “Glass near edge” gets paired with “might fall” often enough that the association sticks. That’s correlation. Physics is causal: gravity accelerates mass toward Earth, and that explanation keeps working when you change the object, the surface, or the planet. Correlation stops working the moment the scene drifts from the training distribution.\n\nThe [CausalPhys benchmark](https://arxiv.org/abs/2606.05966) tested exactly this. They paired 3,000+ physics questions with expert-annotated causal graphs, then checked whether a model’s reasoning chain matched the true causal structure. Result: VLMs routinely produce **plausible but incorrect** answers. The words are right. The machinery behind them isn’t there.\n\nMy favorite concrete example comes from [BilliardPhys-Bench](https://arxiv.org/abs/2605.30900), which tests models on billiard-ball collisions. The authors found a failure mode they call **stasis bias**: when the physical outcome gets harder to infer, models increasingly predict that nothing happens. No collision. No bounce. No motion.\n\nThink about what that means. When a physics engine is uncertain, it still simulates. When a language model is uncertain, it retreats to the safest caption.\n\nGo back to that cup. A photo shows you cup, table, edge, maybe liquid. It does not show you mass, friction, density, or center of mass. A hollow ball and a solid one look identical in a frame and behave completely differently once pushed.\n\nThis isn’t me speculating. The PhysBench team ran an error analysis on 500 model mistakes, and the two biggest causes were **perception errors and missing physical knowledge**. Not sloppy reasoning on top of good inputs. The hidden physical variables were never recovered from the pixels in the first place.\n\nPhysics also runs on numbers, not adjectives. “The ball is rolling fast” isn’t a physics claim. 10 meters per second on a 30 degree incline is. On [QuantiPhy](https://arxiv.org/abs/2512.19526), a benchmark for quantitative kinematic estimation from video, even the strongest current models fail to reach the human average. And humans are just eyeballing it.\n\nVLMs are good at adjectives. Physics needs equations.\n\n(PS: this is why I’m less impressed by “look, it explained the physics!” demos than most people seem to be. Push the scenario slightly outside distribution and the model’s confidence doesn’t drop, even though its accuracy should. Confidence without calibration is just vibes.)\n\nA single frame can’t tell you if a ball above someone’s hand was just thrown up, is falling, or is about to be caught. The pixels are compatible with several different physical histories, and nothing in a standard VLM forces it to pick the one consistent with what happens next.\n\nThat forcing function is exactly what conservation laws give a real physics engine: energy, momentum, and mass don’t vanish or teleport. A caption-trained model has no such constraint baked in, which is why it can describe gravity accurately in one sentence and accept a physically impossible video in the next.\n\nThat is precisely what IntPhys 2 measures. And precisely where models collapse to chance.\n\nLanguage consistency is not world consistency. Only one of them is physics.\n\nKids learn physics by dropping spoons and spilling water long before anyone hands them an equation. You act, the world pushes back, you update. VLMs never get pushed back on. They know the sentence “a heavy object is harder to lift.” They’ve never lifted anything.\n\nThe most interesting counterexample right now is Meta’s [V-JEPA 2](https://arxiv.org/abs/2506.09985). Instead of predicting captions or pixels, it predicts future states in an abstract representation space — learning the dynamics of a scene rather than its surface appearance. Trained on over a million hours of video plus only about **62 hours** of real robot data, it hit **65 to 80% success** on pick-and-place tasks with objects and environments it had never seen.\n\nNote what that recipe is: a world model plus a small dose of embodied experience, beating caption-scale pattern matching. It rhymes with everything I’ve written about [world models](https://pub.towardsai.net/what-is-a-world-model-inside-the-ai-idea-behind-2026s-1-billion-bet-f98986c4d779) and [scientific machine learning](https://medium.com/@farhat.shaikh/scientific-machine-learning-the-ai-that-actually-knows-physics-75daadfef135) on this page.\n\nA VLM can look at a falling ball and say “gravity.” That sentence is cheap. The benchmarks now put a price tag on the difference: near-perfect humans and coin-flip models on physics an infant knows.\n\nKnowing what happens if the ball were heavier, the surface were icy, or the whole scene played out on the Moon, that’s the expensive part. And it’s the part that actually matters once these models start controlling robots instead of writing captions.\n\nUntil a model can pay that cost, it hasn’t understood the ball.\n\nIt’s just described it very well.\n\n**Your turn, DataChefs:** if you could give a VLM one physics test from your own life, what would it be? Drop it in the responses. Best one might end up in a future Recipe. 👀\n\n*If this made benchmarks make sense, follow for more. New Recipes, no buzzwords, just real talk.*\n\n📚 **Sources:** [PhysBench (ICLR 2025)](https://arxiv.org/abs/2501.16411) | [IntPhys 2 (Meta AI)](https://arxiv.org/abs/2506.09849) | [Frontier VLM Failure Modes](https://arxiv.org/abs/2510.06251) | [CausalPhys](https://arxiv.org/abs/2606.05966) | [BilliardPhys-Bench](https://arxiv.org/abs/2605.30900) | [QuantiPhy](https://arxiv.org/abs/2512.19526) | [V-JEPA 2 (Meta AI)](https://arxiv.org/abs/2506.09985)\n\n[Your AI Can Say “Gravity” Without Knowing What It Means](https://pub.towardsai.net/your-ai-can-say-gravity-without-knowing-what-it-means-8ae0d2d034a6) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/your-ai-can-say-gravity-without-knowing-what-it-means", "canonical_source": "https://pub.towardsai.net/your-ai-can-say-gravity-without-knowing-what-it-means-8ae0d2d034a6?source=rss----98111c9905da---4", "published_at": "2026-08-10 16:31:01+00:00", "updated_at": "2026-08-10 16:51:11.908248+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "computer-vision"], "entities": ["PhysBench", "GPT-4o", "Meta", "IntPhys 2", "CausalPhys", "Physion", "CLEVRER"], "alternates": {"html": "https://wpnews.pro/news/your-ai-can-say-gravity-without-knowing-what-it-means", "markdown": "https://wpnews.pro/news/your-ai-can-say-gravity-without-knowing-what-it-means.md", "text": "https://wpnews.pro/news/your-ai-can-say-gravity-without-knowing-what-it-means.txt", "jsonld": "https://wpnews.pro/news/your-ai-can-say-gravity-without-knowing-what-it-means.jsonld"}}