In June this year I joined AMI Labs, a company trying to develop the next generation of AI models that go beyond the limits of the LLMs and image/video-gen models that are ubiquitous today. This post discusses why among all the AI companies out there, I thought AMI's mission and vision were interesting to me, and why I ended up joining AMI to help drive that vision forward.
NOTE: I am not a ML/AI expert, I am an infrastructure engineer. So some of my understanding of these ML/AI concepts may be slightly imprecise or inaccurate
I have never been an ML or AI export. Apart from the Claude/Codex/etc. chat assistants that have become commonplace, my last experience working with "AI" was my college class MIT 6.034 that I took back in 2011. Since then I have been doing web development, cloud infrastructure work, developer tools, programming language work, and open source projects in these domains. Basically everything, except mobile app development and ML/AI.
I did not foresee the AI wave in 2025. I expected the AI chatbots that appeared in 2023 would continue improving at a modest pace. I did not expect that over the course of a single year, I would find myself "never writing a single line of code myself ever again". After being very good at writing code for the past 20 years of my life, it was incredibly disorienting to find large parts of my skillset and professional identity were rendered obsolete overnight. And so I took the time to reset and figure out what the best way was for me to get involved in this ML/AI field that I had neglected for the previous decades.
I was first introduced to the folks at AMI through a mutual friend. AMI was a new AI research lab opening a research and development office in Singapore, which is very rare: most tech companies' presence in Singapore is purely a sales office, perhaps with some forward deployed engineers or solution architects, or engineering to integrate with local partners. Databricks, Stripe, OpenAI, Palantir, etc. all have offices here that follow that model. When I found out AMI was opening an R&D office, that immediately caught my attention.
What made me more interested was what AMI was working on: these "world models". Again, I had no ML/AI background at all and zero idea what on earth a world model was. But learning about what AMI was trying to do felt surprisingly intuitive:
There is a wide range of use cases where LLMs, despite their success in coding and reasoning, do poorly: LLMs generally aren't as good at understanding images, understanding videos, despite enormous amounts data and compute used to try to train them to do so
Image/video-gen models are excellent at generating high-quality still images or hollywood-level CGI, but they still have problems with object permanence, basic physics, and other things that indicate they don't really understand what's going on in the images or videos it generates.
There is a massive breadth of use cases where LLMs and video generation models do poorly or not at all: self-driving cars, robotics, basically anything to do with understanding and navigating the physical world.
We have all been surprised by the brilliance that LLMs and image/video-gen models can show, and we have also all laughed at the absolute stupidity that the same model can demonstrate. Your LLM can give you an explanation of quantum mechanics written in 16th century shakespearean english, and your video-gen model can give you a cool video of Deadpool fighting Darth Vader, but neither can pour a jug of water into a glass or ride a bicycle. What robotics applications exist today in industrial robotics or self-driving cars are fragile one-off systems that require huge amounts of training data and compute to work at all.
What makes AMI interesting is that it's not just scaling LLMs like the rest of the field, but rather developing the Joint Embedding Predictive Architectures, or JEPA.
Traditional LLMs or image/video-gen models are focused on training a model to read and write text or output pixels
In JEPA, your model receives and outputs embeddings, and you train the model to predict or extrapolate how these embeddings will change over time or in response to input
Embeddings are typically much more compact and much more semantically meaningful than raw video or sensor data. An AI model trained on them should hopefully be able to "model the real world" much better than one trained to read/write text or output video pixels, hence the name "world model". Such a model would be useful for the myriad of physical-world use case where LLMs and image/video-gen models fall short.
LLMs and image/video-gen models have revolutionized NLP and image/video generation so much that anyone can trivially pull out ChatGPT or NanoBanana and start doing cool stuff, but AI models that understand the real world through video input or other sensors remain an arcane art hacked together with heuristics and ducktape. In many ways, "physical AI" today feels very much like NLP was 10 years ago: piles of bespoke models stitched together enough to kinda/sorta work. To me it seems that surely there must be a better way,
While JEPA is still an evolving technology, I felt there is a decent chance that JEPA could be the solution to these challenges of physical AI. The first general-purpose models that anyone off the street could ask to pour glasses of water or ride bicycles will be JEPA models, perhaps not now but some time in the near future. In one of my side projects earlier this year, I was trying to count how many people are walking around a retail shop via security camera footage. I managed to get something working, but it surprised me how rudimentary and fragile the state of the art was:
Run separate segmentation/person-recognition/pose-detection image models on every single frame
Wire them together to try and track people across frames with a bunch of hacks and heuristics.
In the end you have a pile of models held together with duck-tape and a prayer to try and approximate a person-tracking system.
As you can imagine, this doesn't work well at all. In this small example clip taken from youtube (https://www.youtube.com/watch?v=-1bRhYjw1qE) we can already see our state of the art YOLO26x/LightMBN/OSNet/OccluBoost-BoxMOT/etc. system fall: the ML system loses track of people constantly as they walk around, and mis-identifies them as new people:
Someone will walk behind a shelf, come out the other side with their body at a 45 degree angle, and the system will think it's a new person. Or two people will walk in front of each other, and then walk apart, and it won't know which one is which. There is no concept of "object permanence", only a frame by frame guess at what persons or objects exist where. No thought "hey this person that we know 100% was here last frame and we only think is 49% likely to be there this frame can't just bloody disappear".
This should not need human-level intelligence - even a cat would be able to perform vastly superior object and person tracking of these scene in real time - but the current state-of-the-art open models and techniques are just unable to do this simple task.
Frontier models like ChatGPT or Gemini do perform somewhat better in this task, and I am sure there are proprietary models and techniques that perform better than the open-weight models and open source techniques I used. But in general the quality of AI video understanding that is table stakes in understanding the physical world just hasn't progressed the same way that NLP or image/video generation have.
ChatGPT and related technologies revolutionized natural language processing making it accessible to every random person off the street via a simple prompt. Midjourney, Nanobanana, etc. did the same for image and video generation. Anyone can train their own GPT model from scratch for a few dollars, or fine tune one of the excellent open-weight models freely available. Frontier LLMs can trivially outperform the smartest researchers or engineers in many tasks (thought not all!), and image/video generation models can sometimes outperform some of the most capable digital artists.
When is the "ChatGPT moment" or "Midjourney moment" going to happen for AI models that can look at a scene via a video feed and actually understand what's in it? When can our AIs move beyond understanding text or generating clickbait videos to operating robots, or driving cars with simple zero-shot instruction or prompt? When can our AI with the knowledge of every PhD subject in the entire world, learn to keep track of people as they walk slowly around a room?
In order to discuss things in english or to generate a plausible looking video, an LLM or image/video-gen model must have some kind of model of the world (a.k.a "world model") encoded in its model weights and activations. But because LLMs and image/video-gen models are trained on reading text and generating text and videos, they also have to spend time on other things:
These conceptual limitations have real consequences:
LLMs are slow with limited context windows because they need to really pay attention to every single input token, which means a 1-million-context LLM taking seconds to respond is many orders of magnitude too slow to ingest 4k video sending you 4 million pixels every 16.6 or 8.3 milliseconds that are mostly un-interesting
Image/video-gen models generate pretty images and videos that don't make sense, resulting in the Uncanny Valley look of much AI content where it is an excellent superficially but just doesn't make sense semantically in a way that humans can often pick up subconsciously. If a model doesn't understand how the physical world works, would you trust it to not crash your car?
Fundamentally, LLMs are very good at reading/writing text, and image/video-gen models are very good at generating images or videso, but for many use cases such functionality is completely unnecessary. In theory, a model without that baggage could be much smaller, faster, and more reliable than a LLM or image/video-gen model in the physical-world use cases like robotics or self-driving which rarely need the capability to write Shakespearean prose or generate clickbait videos.
If I ask you what the scene outside your window is, you are not going to be thinking in millions of words of english text, neither are you going to be generating a mental image in 4k HD video. Rather, you have some abstract idea of what is out there: These kinds of "fuzzy abstract ideas" are how people think. You focus on the core ideas, ignoring things such as:
So if we wanted to come up with "fuzzy abstract ideas" for AI, what would it be? It wouldn't be 4k video frames and it wouldn't be paragraphs and paragraphs of english text.
It would be embeddings!
Embeddings are already the "fuzzy abstract ideas" of the ML/AI world. They already represent the "important" parts of the input image, video, or text, with all the unnecessary stuff filtered out:
Apart from the intuition behind them, embeddings also have many concrete properties that make them nice to work with:
Must more concise than input video frames, e.g. A few thousand 32-byte words rather than several million 32-byte pixels per frame
Much more semantically meaningful than input video frames: all the irrelevant detail in a video frame is filtered out to generate the embedding, so only the important stuff remains
So if I wanted to come up with some way to emulate the evolution of fuzzy abstract ideas that goes on inside someone's head, training a model to work with embeddings rather than raw text/images/video seems like the most obvious, straightforward thing to do. And that is what JEPA is all about.
There's obviously a lot of depth to the idea of JEPA: exactly what embedding models are chosen, how embedding and predictor models are trained, how output decoders are trained, how to prevent collapse. I am not an expert in any of these things: I'm just providing the infrastructure and platform for the experts do their work. But the end result is that JEPA models receiving, predicting, and reasoning about the embeddings will hopefully be both (1) smaller and (2) more semantically effective and (3) more generalizable than than the equivalent LLM, VLM or image/video-gen model, when applied to real-the noisy high-volume data that comes with physical-world use cases.
One specific area that JEPA improves upon LLMs is the handling of noisy, high-volume data. Unlike the english language or coding language tokens that LLMs are trained on, video (any many other real-world data sources) are both noisy and voluminous:
Any video feed is full of noise: camera ISO noise, codec compression artifacts, the vibrations of a self-driving car, or the wobble of the camera-man's hands
These video feeds are also huge: a 4k video produces a 8 million pixels per second, commonly 30 frames per second.
Concretely, this means that LLMs that are trained to focus their attention on high-information text content are not a good fit when you need to shove in huge amounts of low-information video. A model may take a 16x16 patch of pixels as a single token, so watching a minute of 4k/30fps video with 15 billion pixels would work out to 60 million tokens: this doesn't really fit into the context windows of even longest context LLMs. And humans and animals can easily process vastly more than 60 seconds of video!
There are of course a bunch of hacks that you can do - downsampling framerates, resizing the image, etc. - but the underlying limitation is still there. LLMs are just not that great at handling large quantities of noisy mostly-useless input data that many real-world sensors and input modalities provide.
The issue with noisy high-volume data isn't limited to video. For example I have personally found the LLMs woefully inadequate at dealing with things like time-series data. Giving time-series data to the latest Fable, Sol, etc. frontier models, they are often unable to notice things or find correlations that a human glancing at a dashboard can spot instantly. There is a whole world of sensor data out there that is noisy and high-volume, and in robotics dealing with the noisy high-volume sensor and actuator data is one of the greatest challenges of the field.
JEPA, perhaps more than more other model architectures, is good at extracting useful signals and facts from noisy high-volume real-world data. Where LLMs have perhaps "solved" the problem of reading and writing dense, meaningful text, perhaps JEPA could be the architecture that "solves" the problem of dealing with the myriad of real-world use cases where the input data isn't nearly as well behaved.
10 years ago in 2015 I had a friend who spent years working at x.ai (not related to the current Twitter website URL), a startup trying to schedule calendar meetings by understanding and replying to emails. They had piles of ML models and regexes and hacks and heuristics, and eventually they died when it proved impossible to do reliably. I did get a chance look at their code, and in fact it looked very much like the "count people in the security camera footage" project I had worked on earlier this year, just for text rather than for video: a pile of bespoke machine-learning models buried under a mountain of heuristics and regexes.
We used to have separate models for "sentiment analysis", "topic categorization", "profanity filters", duct-taped together with hacks and heuristics that often didn't work (e.g. the famous Scunthorpe problem). Now, we have general-purpose LLMs that anyone can ask to do anything they can imagine (as long as its text related). Physical AI in 2026 feels very much like how NLP was back in 2015: hand-crafted models for "object detection", "pose detection", "segmentation", "inter-frame object tracking", still duct taped together, still kinda-sorta working but not really.
Some physical AI models exist today, but at great cost and herculean levels of effort. A typical self-driving car model is trained from scratch on millions of hours of video data, whereas a human teenager can learn to drive a car pretty effectively in about 10 hours. Even dogs can drive cars! What if we could just take a off-the-shelf general-purpose world model that understands how the physical world works, and fine tune it to drive a car in 10 hours? What if all the cool sci-fi robotics ideas - navigation around a room, walking up stairs, driving vehicles - could be as easy and robust as talking to ChatGPT is today?
GPT-2 was a toy project back in 2019, but eventually became the frontier LLMs we see today. What if the small-scale uses of JEPA we see today - which are already proving very capable at extracting meaning from noisy real-world data - could be the frontier world models of tomorrow?
10 years ago in 2017, I joined Databricks before it was cool: a small no-name company along Spear street in San Francisco. It was so unknown that potential candidates from Google or Facebook or Amazon wouldn't even return our calls to talk to us. I joined Databricks because I thought that big data was one of the most important parts of the burgeoning ML/AI field, based on some fundamental observations from big companies like Google or Facebook published in papers like Hidden Technical Debt in Machine Learning Systems that explored how the vast majority of ML/AI is not the clever algorithms, but the plumbing that surrounds those algorithms, and Databricks felt very well positioned to win that space.
Now I joined AMI for a similar reason: I think that what they are trying to do seems very... reasonable and intuitive. LLM and image/video-gen models are without a doubt amazing, but with many known pitfalls weaknesses. If I think about how I think, and I look at the different AI model architectures out there today, JEPA seems like the closest thing we have to a "human-like" intelligence. An AI that reasons and thinks in abstract concepts, rather than in millions of words of english text or in 4k HD video.
There remains the challenge of proving that all this stuff actually works at scale, but if it does then AMI and JEPA could potential solve a lot of the weirdness, idiosyncracies, or limitations that come with LLMs and image/video-gen models today. And not by piling on more hacks-on-top-of-hacks, but by using a fundamentally architecture that actually reflects what "intelligence" is all about.