Why Video Agent models are next — Ethan He, xAI Grok Imagine

Ethan He, an engineer at xAI who built Grok Imagine in three months, argued that video models derive their intelligence primarily from large language models rather than video training data, and predicted the next frontier for generative media will be video agents capable of planning, editing, and iterating across creative tasks. He suggested that the evolution of video generation will mirror AI coding's shift from one-shot output to multiturn reasoning and orchestration systems, with real-time, interactive world models depending more on language models and agents than on diffusion alone.

We’re announcing AIEWF speakers this week Take the AI Engineering Survey Today’s guest Ethan first joined us for the LS Paper Club as the lead on NVIDIA Cosmos World Model https://www.youtube.com/watch?v=og59L4JECz4&pp=ygUWbGF0ZW50c3BhY2V0diBldGhhbiBoZQ%3D%3D , but then joined xAI and built Grok Imagine in 3 months: He comes back on Latent Space with some nuclear hot takes: that Video Models primarily get their intelligence from LLMs , not from training on video data, and that the next frontier for truly interactive, realtime, long-horizon world models is to work on LLMs perhaps Interaction Models https://www.latent.space/p/ainews-thinking-machines-native-interaction as well… Put it this way: In the near term, the next Sora won’t be a better video model, but a video agent . Generative Media may more closely follow the evolution of AI coding which went from focusing on one-shot output performance and cost, to multiturn reasoning and planning models for agents and systems that can plan, edit, test, debug, and submit PRs. At a certain point, coding models got so good that the only significant next step to improve performance was handling the orchestration of these models. Now as the performance of video models increases significantly across realism, consistency, & prompt adherence while becoming more cost efficient, the next evolution of video generation may also be systems that can plan, generate, edit, critique, and iterate across an entire creative task. In this episode, Ethan joins swyx and Vibhu to unpack what it actually takes to build frontier image and video systems : data, VAEs, diffusion transformers, audio-video alignment, inference speedups, and the hidden cost of storing and moving massive video datasets. From building NVIDIA’s Cosmos world model to joining xAI as was being built from zero to one, Grok Imagine https://grok.com/imagine Ethan He has been at the center of some of the most important work in video generation, multimodal models, and real-time world models. We go deep on Grok Imagine , how a small xAI team shipped its first multimodal video model in three months , why iteration speed matters more than almost anything in model development, and why many of the biggest gains come from fixing tiny bugs in data and training pipelines. Flipbook: The future of Videomaxxing Video agents are almost a sure bet to be the trend in the coming year. We end with a glance at what’s beyond video agents: Flipbook caused a minor sensation this year when it was released, but most treat it as a fun demo. Ethan takes it very seriously — with the speed and cost of inference coming down every year, the future of custom video JIT UI is closer than you think. We talked about why videogen models may become the front end of AI, how generative UI could replace traditional HTML/CSS , why world models need to be real-time, interactive, and long-horizon, and why the future of video generation may depend more on language models and agents than on diffusion alone. We discuss: Why fast iteration mattered more than meetingsWhy small training bugs can drive huge model quality gainsWhy coding models may make compute the bottleneck againHow image and video models are trained with synthetic captions The role of VAEs and latent space in frontier video modelsWhy image models are the foundation for video modelsThe tradeoff between temporal compression and real-time interactivityWhy future interfaces may go from user intent to pixels The hidden cost of training video models: storage, egress, and GPU hours How step distillation and consistency models like OpenAI sCM https://openai.com/index/simplifying-stabilizing-and-scaling-continuous-time-consistency-models/ makes video inference orders of magnitude fasterGrok Imagine 0.9 and large-scale audio-video generation Why audio-video alignment is harder than text-video alignmentEthan’s definition of world models Reference-to-video, video extension, and long-context video generation Why xAI’s research communication undersells Grok Imagine How xAI culture shaped the speed of developmentAI watermarking, SynthID, and detecting generated media Why prompt rewriting matters for video modelsGrok Imagine Agent and the rise of video agents Why language models may unlock better video generationRobotics, physical AI, and embodied world models Why Ethan left xAI and shifted focus toward LLMsSelf-managed context, memory, and the next frontier for language models Ethan He Timestamps 00:00:00 Introduction 00:01:25 From NVIDIA Cosmos to xAI 00:03:24 Building Grok Imagine from Zero to One 00:10:07 How Image and Video Models Are Trained 00:18:53 Video Compression, VAEs, and Real-Time Tradeoffs 00:22:10 Generative UI, Flipbook, and Neural OS 00:32:10 The Cost of Training Large Video Models 00:37:04 Distillation, GANs, and Fast Video Inference 00:41:21 Audio-Video Generation and Grok Imagine 0.9 00:48:34 What Makes a World Model? 00:55:51 Reference Videos, Long Context, and Video Memory 01:00:11 xAI Culture, Research, and First-Principles Building 01:09:45 AI Safety, Watermarking, and Prompt Rewriting 01:13:10 Video Agents and AI-Assisted Creation 01:27:32 Why Language Models Unlock Better Video 01:31:15 Robotics, Physical AI, and Embodied World Models 01:32:38 Why Ethan Left xAI 01:34:16 Self-Managed Context and the Future of LLMs 01:38:43 Ethan’s Career Path and Closing Thoughts Transcript Introduction: Ethan He, Latent Space, and the Path to xAI Swyx 00:00:00 : We’re here in the studio with Ethan He, most recently of xAI. Welcome. Ethan 00:00:10 : Thank you. Glad being here. Swyx 00:00:11 : We’re also here with Vibhu. you were first coming to us or joining the latent space world because you were working on Kosmos at NVIDIA, and you did a paper. We loved it. you presented it as well, so thank you for doing that. Ethan 00:00:23 : I’ve actually, I also presented the MoEs twice at latent space. Swyx 00:00:29 : How did you actually hear about us? Did we reach out to you? Is that how it worked? Ethan 00:00:33 : No, actually, I-- the community. Like I realized, oh, there is this online community that people talk about AI and also learn from each other through papers every week through the Paperclip. It’s very nice. Ethan 00:00:49 : I learned a lot. Swyx 00:00:49 : I think three years stop. We haven’t stopped even on Christmas and New Years. many weeks I want to stop but it keeps going. Vibhu 00:00:58 : No, that was good. I think you had posted that you worked on a paper, and I was “Oh, very cool. We have Paperclip. Present then.” Vibhu 00:01:04 : But I might have reached out to you after. Swyx 00:01:05 : you-- because it’s an amateur club, right? Swyx 00:01:08 : so it’s very unusual and but we have sometimes paper authors come by and actually explain the paper. Today we just did, the poolside paper, which was apparently very good. Vibhu 00:01:18 : Came out yesterday. Vibhu 00:01:19 : pretty interesting, right? Fully open. They talk about everything, systems. So it’s a good one. We’ll, we’ll recommend people to read it. Swyx 00:01:25 : Bring us up to speed on your transition to xAI, ‘cause I actually don’t even know when you joined. just like tell the, tell the story about the sort of transition. From NVIDIA Cosmos to xAI: Scaling Video and World Models Ethan 00:01:34 : Before xAI, I was working on Kosmos world model as in-- at NVIDIA. So Kosmos is, it’s a giant video foundation models that can-- that aims to simulate the world and for-- it serves as a foundation of-- for all of the roboticists to build on top of. There, once I built the Kosmos one, I realized as this thing also has a scaling law similar to language model, we need to scale up the video models further. that’s, that’s why I realized I need to move to somewhere with much more compute resources. That’s how I Swyx 00:02:13 : Than NVIDIA? Vibhu 00:02:14 : The GPU rich came themselves. Vibhu 00:02:19 : And timeline-wise, when was Kosmo? It was pretty early, right? It was open world model, open paper, everything. Ethan 00:02:25 : It was end of twenty-four. Vibhu 00:02:28 : End of twenty-four. Ethan 00:02:30 : Then at mid twenty-five, I moved to xAI. At that time-- I joined about the time when xAI was about to build video models and in multi-model models. There were no infra, no data, and no model, and it just-- as a few engineers, we built it in three months and released the first model, Grok Imagine zero point nine. Ethan 00:02:55 : And since then, I keep working on video models and move more from training and to post-training of the video models. For example, like a reference to videos, kind of like the cameo feature and, video extensions. And, before I left, I worked on a world model, leading a small team to focus on the real-time long horizon video generation. Building Grok Imagine From Scratch in Three Months Swyx 00:03:24 : Can you give like a rough roadmap of okay, you’re on a brand-new team. Grok previously was only text, or they partnered with BFL for their image gen stuff. What do you-- what are the building blocks, right? You have compute, data you can procure somewhere. Like just what are like the sequence of things that people should think about when you’re setting up a new team? Vibhu 00:03:43 : actually even deeper, not just data you can procure. You guys had to go through getting the data too, right? So you shipped it pretty fast, but yeah Swyx 00:03:51 : three months is like Vibhu 00:03:52 : From everything Swyx 00:03:52 : actually like very surprisingly fast. Ethan 00:03:55 : One thing I say like thanks to my experience at NVIDIA, ‘cause first time when we were building Kosmos together, we built it, for about a year. So this is like the second time I do it. Roughly have an idea, what to do. I say the most important thing is the talent. Everyone were very strong and clever, very close with each other towards a common goal. So that speed up things a lot. So you reduce the communication bandwidth among people, and everyone can work towards the same goal. It’s, it’s like every day there’s not that much meetings on the calendar, like maybe like a, like a sync a day, and after that it’s, it’s just all building. It was pretty fun at that time. Ethan 00:04:47 : And another thing is that xAI has very strong foundations of like data inference, model inference, and the supporting there can help the model develop a lot. When I look at, training models, I don’t so actually the top important thing is like how many, how many iterations can you do, per day? and the more iteration can you do, you can, you can train the model much faster. So if you have very strong infra and you have a lot of compute, you can, you can train these models in very short period of time. That can give you a much larger buffer to, for errors, and it also gives you the opportunity to spot more bugs. Iteration Speed, Compute, and Debugging Model Pipelines Swyx 00:05:46 : What is an iteration? Is it like a few hundred steps or what are you Ethan 00:05:50 : Let’s say just the train-training the model, like from acquire new data and maybe design new algorithms and train a new model, maybe at smaller scale or Swyx 00:06:01 : So cycle time for like any hyperparam that you’re searching. Ethan 00:06:04 : Cycle time and tune to like eval this model. Is this model better than my previous iteration? Ethan 00:06:11 : So Swyx 00:06:11 : So it’s like before you, someone had already set this up that you can iterate very quickly. Ethan 00:06:15 : I think the foundation there is extremely good forDeveloping and research models. Ethan 00:06:23 : And often I find is it-- this is kind of boring, but like a lot of the improvements does not come from new algorithms. It comes from finding small bugs here and there in the data pipeline, in the, in the model training pipeline. Those give, those give the biggest boost to the model quality. Vibhu 00:06:46 : It’s interesting, right? So you say it’s like small team, less communication bandwidth, but also a lot of quality is like find little bugs. It seems counterintuitive, right? You have a lot of people, you can iron out more of those, but it’s interesting to see the other side, right? Swyx 00:07:00 : I also wonder, have you-- do you try using LLMs to look for bugs? I don’t know. Ethan 00:07:05 : I remember at that time it was mid two thousand and twenty-five, so it’s the coding model wasn’t quite there yet. I remem- I remember like December two thousand and twenty-five, it was extremely good. Yeah, I’ve been, I’ve been using it at that time. It’s, it’s helpful. sometimes it produce codes that are kind of difficult to maintain, even though like the first time it built something extremely fast. But it gave the, like a spaghetti code, thousands of lines that I couldn’t maintain, and the LLM itself couldn’t figure out what’s, what’s wrong and how to improve on top of it. But now I find it much better. Yeah, I want to bring up another point here is now coding models are much more efficient and can help us implement stuff much faster. Compute might become a bottleneck again because previously, like if you want to train a new model, say you want to generate new synthetic data and then or write a new algorithm, it might take a few weeks. And during that period of time, you don’t-- you might not have experiments to run. But now you can build that thing within a few hours, then you can immediately train a model. Ethan 00:08:24 : Now you have to have enough compute to try all of the ideas. So compute might be the bottleneck of iterating speed again. Swyx 00:08:36 : yeah, I actually, honestly, I think it’s like kind of a stressful job because you’re “Well, I should be trying everything, and if I’m not, then I’m not doing my job well.” Vibhu 00:08:48 : there’s also the stress of you’re eating thousands of GPUs per hour, which is very expensive and, compute can go to other researchers. Swyx 00:08:56 : You got the daddy Elon to Vibhu 00:08:57 : You got daddy Elon. Ethan 00:08:59 : It was Vibhu 00:09:00 : But there’s still finite amount of compute, like you want to use it, you want to use it well, you want more of it. Ethan 00:09:06 : That was quite stressful indeed. Yeah, I think one thing is the-- with coding models now, like a lot of these jobs can be automated, which is much better. A second, it’s a, it’s a marathon, so you got to maintain good health and, a regular schedule. Vibhu 00:09:28 : It’s, it’s hard to hear that when you shift from zero to nothing in two months. Swyx 00:09:32 : and, I think obviously the culture at xAI is very famously, people work very hard. one thing I did want to dive into, in our-- in the notes that you, that you sent ahead of time, you had specific comments about the cost of Video Gen training. presumably this is on the Colossus-1, right? the two hundred megawatt cluster. Any whatever you want to just share on that. Vibhu 00:09:54 : I think there’s, there’s three things we’re talking about, right? So there’s Video Gen, there’s also the Image Gen model that you put out. Do you want to like complete the, okay, so zero to one, you have a few months. Just what are the stages of create Image Gen model? Swyx 00:10:06 : Oh, yeah, maybe I got distracted. How Image and Video Models Are Trained: Synthetic Captions, Tokenizers, and VAEs Vibhu 00:10:07 : Sorry. and then, from there’s Video Gen, there’s Audio Gen. Would love to get into those next. But what is that first few months like? So small team, a lot of bugs, iterations, but what does it look like? Do we take something off the shelf? Do we just get data compute? What’s, what’s the few months like? How do you go to state-art Image Gen model? How do you just start? Ethan 00:10:28 : I cannot comment specifically how xAI did, but it’s, it’s a quite standard process. I can draw some, examples from Cosmos. So mainly it’s building a video model, you actually need to build a image model first. And building these two models, the data you need is a hundred percent synthetic pair of language and image or language to video. Because on the, on the internet, actually, the videos don’t naturally associate with text. So you can say, oh, like on YouTube, you have the title and you have the description and the comments Swyx 00:11:11 : Title Ethan 00:11:11 : of a video, but usually they’re not relevant to the video itself. And say maybe like the video is a natural scene of mountains or something, and the title is, I’m so happy today. Ethan 00:11:26 : So they have they have no correlation at all. So the first step is to, you have to generate synthetic pair of language with the videos. So you gather videos from the internet, and you use a VLM to caption the videos. So that part, here’s a question, like how do you, how do you gather VLM to begin with? So if there’s no Swyx 00:11:55 : You, so you fuse the model, right? Like Ethan 00:11:57 : Say if there’s no like VLM exists, like how do you generate the text to the beginning, right? It’s, it’s impossible. Swyx 00:12:04 : I see. Ethan 00:12:05 : In the beginning, it’s like you ask human to describe the video as detailed as possible.For example, you ask them to describe everything, like all objects, all characters, and all interaction and dialogues in the, in the videos. So that’s in the protocol of Cosmos labeling. We require the objective we give to the labelers was that you have to describe the video as detailed as possible, such that a blind person hears a blob of text can reconstruct what the video is like from their head. Swyx 00:12:43 : Video or image? You’re talking about images. Ethan 00:12:44 : Video or image, either one of them. Vibhu 00:12:47 : This was pretty common when we went from clip and DALL-E, right? Vibhu 00:12:51 : It’s all training on really detailed captioning of images. So same is applied to video, but instead Ethan 00:12:57 : same applied Vibhu 00:12:57 : of using multimodal model to pass in video images and write rich descriptions, you can also Swyx 00:13:04 : I think there’s this traditional perspective of supervised, or, very highly human curated thing. I feel like there’s a unlock with unsupervised, right? Where like you have enough to bootstrap that you can just throw common corpus on it or, whatever. like unsupervised vision and language pairing, right? Like where you just have, interspersed image and text and it just learns. To me, that is the VLM breakthrough that is different from the clip, different from the LM era. Ethan 00:13:36 : It’s interesting to see that you kind of need both data. Ethan 00:13:41 : For example, for the Swyx 00:13:41 : You need it to bootstrap it up. Yeah Ethan 00:13:43 : for the generative model training, there’s also usually like a small percentage of unlabeled data. So the model is instructed to generate a video without any text instruction. That can also help the model generalize. So after this stage of generative synthetic pair, so, one important common step is to train a compressor or a tokenizer of the image or videos. So because, if you train-- If you can technically, theoretically train image or video models on pure pixels, but the problem is that the, it’s, it’s a lot of tokens. So like one image, it’s, a thousand by a thousand, it’s like one million tokens, one million pixels. It’s impossible to train transformer on that. So it’s, you need to train a tokenizer, which can go from image to latent space and latent space back to image. Swyx 00:14:45 : That’s why we named the podcast. Swyx 00:14:48 : But, basically, you’re talking about vocabulary science. Ethan 00:14:50 : so vocab. Swyx 00:14:51 : And so, what is, what is imp-- like a million is impossible? Ethan 00:14:54 : In generative models, the vocab is continuous. It’s a continuous space. We can think about like you map an image to a vector. It’s a, it’s a fixed length vector. It’s sixteen or forty-eight, something like that. And then you map that vector back to the image space. And the mapping is, has-- The mapping is patch-based. So you say you have Ethan 00:15:22 : a sixteen by sixteen patch and you match, you map that patch of pixels into this latent space. Swyx 00:15:29 : We’ve covered this Vibhu 00:15:30 : This is like the vision transformers Swyx 00:15:32 : VAEs, Ethan 00:15:33 : VAEs. Vibhu 00:15:34 : You basically compress your input, you do your generation, you’re reasoning all that generation in smaller dimension, and then you project back out. Swyx 00:15:43 : VAE is a form compression, but I think the for me, the patching thing is from VIT, right? Ethan 00:15:48 : You can make those. Swyx 00:15:49 : Literally the, yeah, the paper is titled like sixteen by sixteen is all you need. something like that. and then I think also, people make a lot of comparisons with this kind of patching with convolutions. Swyx 00:16:02 : Which is you’re, you’re kind of re- reconstructing the old paradigm with the new. Ethan 00:16:05 : Actually, in VAEs, there are, there are both convolution networks and transformers. You can actually do both. Ethan 00:16:14 : After this VAE, so what you’ve got is you’ve got latent space tokens and you’ve got the language tokens. So now the training of the diffusion transformer, usually generative models use diffusion transformers. It is actually quite standard. It’s, it’s very similar to how you train a language transformer models. It’s not that much difference. It’s just the tokens, the visual tokens in, visual tokens out. The only difference is there’s a denoising process. So you train the model to unmask some of the noise. So you add, you add random noise to the visual tokens, and then you train the model to remove those noise to generate the clean tokens. Any inference, the model can iteratively remove noise from a hundred percent noise. Swyx 00:17:12 : And then there’s also, to speed things along on the tech tree of diffusion, there’s CFG, and then there’s, there’s also, latent diffusion that, there’s, there’s someone in there. I think, somewhere along the line, obviously, like stability and all these other guys, pioneered a lot of this, architecture. I don’t know if you want to get into that or just, or do the video side up to you. Bootstrapping Video from Image Models and Temporal Compression Ethan 00:17:37 : After you train such model, such image model, the reason it’s a, it’s a foundation for video models is that image models are cheaper to train, and they have much denser connection between language and text. So, sorry, language and images. For example, you train a billion, you train on a billion images, and there’s a mapping from the text to the image. And the cost to train the same, like the, a billion, a billion text to a billion videos, that’s much more expensive because videosNaturally have more tokens than images. Because the diffusion models, their understanding of, language purely come from this mapping. So if you don’t have enough mapping, so if you only train on like a ten million videos or something, there-- you might not see enough language tokens in your training, so your model does not understand human intention enough. So that’s why you really-- you train-- you first train this image diffusion models, and then you bootstrap the video model from there. Swyx 00:18:53 : One thing I did want to ask, because I-- actually, I think you’re, you’re the first per-- video model person I’ve ever talked to, I think. we’ve, we’ve like talked to Luma and all those folks. There’s all these tricks in video compression where basically frame by frame there’s not that much difference, so actually you don’t have to regenerate or save the whole frame, right? but I think MP4 compression or something else like that. Swyx 00:19:16 : is it tempting to use that? Or as far as I can tell, everyone just treats it as, “No, we would just generate every frame.” Is that roughly the state-art? Ethan 00:19:27 : There are a few different approaches. Let’s say first, like you want to just directly use MP4 compression and use that as the tokens for the transformers to train, right? So people actually have tried that, but the main challenge is the latent space for the MP4 tokens were not, were not very comprehensible for the models. It’s, it’s extremely hard to train on that. And there’s a Ethan 00:20:01 : So that’s why they created VAEs, which creates more continuous, latent space, so the models can understand that latent space and learn from it much easier. Even within the VAEs, there are different difficulties of the latent space. So you can imagine something the simplest, the most naive VAE is like you have an image, and you just shuffle all of the images into a, into a vector. So you don’t need to train any VAEs, right? But that latent space is extremely hard for models to train on top of. That’s why there are some debate on like how do you compress the tokens. So you mentioned like you can compress frame by frame. Also, you can compress, the temporal dimension. Ethan 00:20:52 : The difference is if you compress the temporal dimension, you get a much higher compression rate. Because there’s temporal redundancy between frames, because, this frame and the last frame, likely they are mostly similar, so there’s only some small difference. for example, I think in 12.1 VAE, they have like a eight by eight by four compression rate. So the four temporal tokens are compressed into one tokens. That can save a lot of, save a lot of the context length. If you do it frame by frame, you have to do maybe like eight by eight by one. Your context length will be four times larger. That being said, the benefit of the frame-- per frame compression, we might come back to this later, is, real-timeness and interactivity. ‘Cause if you, if you strain the output of the model, frame by frame, you can-- the model can respond to any user request immediately. So if you have like a temporal four compression, four times compression, then Swyx 00:22:06 : It might be laggy Ethan 00:22:07 : there’s a lag there in nature. Swyx 00:22:10 : So you’re very pilled on this. let’s just go ahead and bring it up ‘cause we have the visual prepared anyway. There’s some frontier applications of real-time video gen. So Flipbook is one of the examples that went viral recently, right? What is Flipbook? Real-Time Generative UI: Flipbook, Neural OS, and Diffusion Front Ends Ethan 00:22:23 : Flipbook is kind of like a web brow- web browser. You can see like it has the web bro- browser UI on top. The difference is all of the UIs are generated by generative image model in real time, and anything here are fake. But you can, you can explore inside this wor- this imaginary world. Say like we-- here we have engineering the Great Pyramid. Like the model generates this for us to understand how it works, and if we want to navigate around and understand further, we can click on some of the, some of the description here, and the model will generate a new page, new subpage describing the details we want to know about. Swyx 00:23:14 : So it’s basically kind of we’re playing a video, but it’s pausing for our next interaction, and then it just plays the next thing based on our interaction. Swyx 00:23:23 : Which is kind of cool. Vibhu 00:23:25 : and you kind of decide your story. So this was, how do you make a pyramid? levering technique seemed interesting, right? It shows how do you take Okay, I want to know what is this Swyx 00:23:35 : The demo, the demo tweet had more animation between frames. Vibhu 00:23:38 : I think it’s just skipping, Swyx 00:23:39 : Oh, it’s just skipping a lot of frames. Ethan 00:23:40 : they also have a video mode Vibhu 00:23:42 : It takes a lot. There’s a lot of people Ethan 00:23:42 : but, a lot of people are using it. Ethan 00:23:45 : So it’s not available. Vibhu 00:23:46 : There’s a live video stream. We can try, Swyx 00:23:50 : So this is an example of the kind of future that you see at the extreme. We don’t-- we’re obviously not in it today. Swyx 00:23:56 : But in a world where inference is completely free this is better than generating code and text? Ethan 00:24:02 : So this is, this is a final state of where Viva will be at for word model, I think. Imagine internet doesn’t exist, and then you type in google.com. Like what should, what should, what should a model show you?the model can imagine something, and this is what the model imagine. And these web pages, they completely do not exist. So I think as the inference costs come down, we are going to have generative UI for everything. If you think about how the coding model works, so they write code for a web page, and they render the code might be con- converted into binary, and the binary render the pixels on the screen. So we in machine learning, every time we have some breakthrough, obviously it’s, it’s more intuit. So why don’t we have like user instruction to the pixel directly? So the generative UI will be user intention to the pixels directly. And say like even if I want email, let’s say everyone have the same interface, but I want, I want it slightly different. I want the email to show to me like a TikTok, so I can swipe left and right for the emails. And or maybe you want something else. We can have completely different things. Or like I have I’m looking at, Instagram stories, and I don’t like the Like button. I always may click it. And, generative UI resolved it. So it’s going to be a revolutionary replacement of the interface. So in the future, we might have much more powerful Ethan 00:25:50 : LLMs and coding models running behind the scene. And in the, in the front-end, the diffusion model will actually be the front-end to show stuff to you. That’s how I imagine it. Swyx 00:26:02 : Diffusion front-end, deterministic back-end. Swyx 00:26:04 : Something like that. I find that very expensive, but, Vibhu 00:26:08 : I find it interesting you called LLMs writing code on the back end deterministic, but okay. Swyx 00:26:14 : you write it once Vibhu 00:26:15 : Compare it to Swyx 00:26:16 : And then you execute. Ethan 00:26:17 : If you think about the cost, say, let’s say H100 costs $1 per hour, and if you use this eight hours a day and thirty days, so, every month you’re paying this two forty, you’ll actually not wanna pay for that. That’s even more expensive than Cloud Code Max. But if you think about the compute costs come down like two times every year, and I think the future will likely arrive like within few years. Vibhu 00:26:49 : It’s everything, right? compute cost comes down, compute gets faster, model gets smarter Ethan 00:26:54 : More efficient Vibhu 00:26:54 : model gets smaller. Swyx 00:26:55 : I don’t know why you say two times, ‘cause I think it’s like 100 times. In language models, it is roughly one hundred to a thousand times every twelve to eighteen months, for the same given level of LMSys, ELO. Vibhu 00:27:08 : That’s a net of everything, right? That’s model performance alongside compute. So different than just compute costs come down. But, a very interesting future. Swyx 00:27:19 : So the web designers will have to shout out that accessibility is an issue, right? how do you deal with screen readers or whatever. But yes, this is higher bandwidth storytelling than anything you can possibly generate with code, right? So I think that’s the rough idea. Ethan 00:27:34 : And I’d like to add a little bit that so human naturally have the maximum bandwidth when we are looking at things, look at videos, and we also have maximum output bandwidth when we are talking. So in the future, it might be something like we talk to AI models, and the AI model responds back with a generative UI. So that would be the maximum input and output bandwidth to interact with AI models before neural link happens. Vibhu 00:28:06 : And it’s also very custom, right? Some people are very visual, some people are not as visual, right? They prefer the text. But the best thing about generative UI, right, it can also be text. Swyx 00:28:17 : There’s another project that we wanted to highlight, which is the Neural OS. Kinda similar idea, but here you’re literally operating, simulating an operating system with a video model. Swyx 00:28:27 : and you can play Doom, you can do Firefox. I find this like mildly less impressive, obviously, because it’s an OS that I can run. Swyx 00:28:37 : But here everything is imagined. Vibhu 00:28:40 : I was, used to the Command+W to close the Firefox tab. It didn’t crash. That’s why I said Swyx 00:28:45 : It’s too immersive. Vibhu 00:28:46 : It’s, it’s too immersive for me. Swyx 00:28:47 : Too immersive. Vibhu 00:28:48 : I wanted to close the tab. Vibhu 00:28:49 : But yes, I can play generated diffusion. Swyx 00:28:51 : this is shockingly fast. Swyx 00:28:54 : Because I remember there was a demo about like maybe one to two years ago. Someone tried to do the first-person shooter with a image model. There was no consistency. It was very slow. But here it looks like realistically it’s-- this is Doom. Vibhu 00:29:07 : I think there’s two sides to that, right? There’s okay, what is running a game? The heavy part of it is actually the game engine, all the lighting, all that stuff, the graphics. This is just kind of video, right? Like we’ve solved consistency. This is still, it looks like a few years old image generation. There’s some temporal consistency, but it’s, it’s kind of just images stitched together as frame video. But it’s a good visual representation to pi- to picture the future you wanna see, right? that’s, that’s what I see in these more so. Ethan 00:29:38 : This reminds me of how the video models gets better and better. So Neural OS is kinda if you just look at it feels like it’s just a crappy version of the, like the Windows we could have, right? And, but the difference is, so the model, this model is overfitted on the existing operating systems. It can generate nothing different than that. But it’s actually also similar to video models. So when we are training these video model, image model, we train them on internet. There’s no imaginary supernatural stuff on the internet. But once we train this model, you can prompt the model to generate something supernatural that have never existed in the data set. So if you train your Neural OS or neural computer on the standard screen recordings on the entire internet. The model can imagine completely new interface to interact with the computer. Swyx 00:30:43 : This is one of those things that is magical to me. usually generalizing out of distribution is bad, but somehow we have learned some kind of internal world model that you say, this plus, but it looks like rainbows and butterflies, it’ll do it and it will kind of make sense. Swyx 00:31:03 : So yeah, that’s kind of cool. Yeah, I don’t know if there’s any comment more on there. I do, I do wanted to, I did wanted to touch a little bit more on the model architecture stuff, which I think you were getting. It’s, really fascinating. We don’t get a chance to talk about this enough. So one of the papers that we covered, we’ve covered every annual, segment anything release. and I don’t know if you follow-- you’re a computer vision guy, so you Ethan 00:31:26 : I know Swyx 00:31:27 : . So they did memory attention, which is kind of interesting. And I always think, anything where you can, across the temporal dimension, keep some consistency, I think it’s, very fascinating, and I don’t know if Basically, does that-- the CV side bleeding into video gen side, I think is underexplored, right? we talk about it for labeling, but actually you can borrow the architecture itself. Ethan 00:31:50 : There’s, there’s also complete different approaches, right? you brought up the term world model, so we went from video model to world model. There is diffusion, but there’s also other approaches that people are doing. So maybe we get into those after as well,? Swyx 00:32:03 : He has a whole definition of world models and stuff. I feel like we threw a lot at you. Whatever you want to comment on. Why Video Models Are Expensive: Storage, I/O, and Training Scale Ethan 00:32:10 : I think one thing that we should actually comment back on is okay, so we were talking about the steps to train image gen to video model. One thing we don’t see as much of is okay, you brought up the delta in training data, right? So Ethan 00:32:24 : you won’t have as much a video model might not generalize, but what is the cost of training a large video model? So we know for LLMs roughly, okay, even like the poolside thing that came out today, right? It’s a Gemma level model trained on roughly forty trillion tokens at this many H200s over this much time, right? You can see what is the exact cost of that. So how many GPU hours over how much H200 costs? So how do we do the back-end math of, same thing for video models, image models. How do you, how do you kind of break that down? I can share some back-envelope calculation. So surprisingly, video models is-- the cost is very-- is comparable to language models and obviously the largest scale is language model, maybe like a medium scale to language models. I said just storing the videos alone, it costs a lot. You can, you can maybe look up on AWS or something. Ethan 00:33:20 : You really, say if you have a billion videos and let’s say, let’s just say like each video, like five megabyte, then you need five petabyte to just store those videos. And also remember we talk about you use a VAE to compress the videos, and you also need to store, typically you need to store those continuous feature, in-- also in your storage. That’s also comparable size with the videos themselves. So just storing these videos and the features is tens of petabytes alone. And, Swyx 00:33:58 : I just, I just looked up the calculation. Five petabytes on S3 Standard is one hundred K per month. Ethan 00:34:05 : And Swyx 00:34:05 : It’s comparable Ethan 00:34:05 : and you need Swyx 00:34:06 : And Ethan 00:34:06 : And then like tens of petabytes, two hundred K. And even more expensive is you have the ingress and egress. Swyx 00:34:13 : Oh, yeah. Ethan 00:34:14 : Like you-- through the internet. You have to just to download those videos, I believe it’s, it’s more expensive on AWS than just storing those videos. Swyx 00:34:25 : Storing, yeah. Ethan 00:34:25 : And each training runs, you probably need to pull them once. If you train multiple times, it’s, it’s even more than that. So it’s like just storing the network, those costs is just, it would be a few, a few millions per month to just storing everything, not to mention the GPU cost. Ethan 00:34:45 : And Swyx 00:34:45 : my side tangent, the compute rental, like GPU rental is very efficient. There’s one side, okay, you can be XAI and build your data center. Should we not just build our, storage compute as well? Like Ethan 00:34:57 : Of course Swyx 00:34:57 : cloud cost compared to just, Ethan 00:34:59 : You save so much Swyx 00:35:00 : store. Yeah, exactly. Swyx 00:35:01 : Especially with like egress and stuff. So. Ethan 00:35:04 : That’s a good idea, but it also comes to-- there are some of its own challenges. Swyx 00:35:09 : Of course, of course. Ethan 00:35:10 : like people who build the GPU data centers, they might not expect this much, storage. And yeah, people build storage, typically they just build it somewhere with just CPUs. Swyx 00:35:23 : I just looked it up. Five-- AWS only charges for egress, not ingress. Tier five for five petabytes is two hundred and thirty K. Ethan 00:35:32 : Even more expensive than the storage. Swyx 00:35:34 : But storing is per month, right? You check in, then you cannot check out. so it’s so cool. It’s okay. So there’s that side. Ethan 00:35:41 : So the TLDR, my backhand math Swyx 00:35:42 : Data is larger than you think. Yes. Ethan 00:35:44 : my backhand math of GPU hours times GPU cost is also very much, I’m missing some storage. Swyx 00:35:49 : You’re also-- you’re basically like also more IO bound than normal training. Swyx 00:35:55 : Yes. ‘Cause like data loading, so caching everything, it becomes super important. Ethan 00:36:00 : So in Cosmos, we did a lot of optimizations to make it not IO bound. So, speaking of the training, actually training the model, the GPU cost, if you look up like the open source model, how big these video models are, I think like LTX has nineteen B parameters. That’s a dense model. And people are also exploring, MoEs, so it might be twenty B active and, like a hun- hundreds B, total. So that’s, that’s even-- that’s similar size as medium-sized LLM models. And if you, if you look at number of tokens-Uh, we disclose that in Cosmos. It’s also like tens of trillions of tokens on the visual tokens. So putting this together, the cost of, training these video models, it’s actually comparable with LLMs. Not to mention, the infra is slightly different from LLM, so it might be less efficient to train these models. Inference Speedups: Step Distillation, Consistency Models, and GANs Swyx 00:37:04 : Do you get the benefits of traditional diffusion speed-up? So for, images, there’s LCM, LoRAs for, fine-tuning. There’s, there’s a lot of stuff that’s been Ethan 00:37:15 : Flow matching. Swyx 00:37:16 : there’s flow matching. There’s a lot of stuff that’s been done. there’s some overlap that applies to diffusion on the inference side and stuff or? Ethan 00:37:23 : so the difference-- the inference side is a completely different story. Ethan 00:37:28 : I think for the training side, it might be a little bit hard to reduce that cost. And for the inference side, the biggest gain is from the distillation of these models. You can-- It’s called step distillation, slightly different from knowledge distillation in LLMs. So you-- Typically, for flow matching models, you need like 100 steps or something. Like a distortion model even need even more, like 1,000 steps to generate a good image or video. A step distillation is try to learn to generate fewer step from the model itself. It’s kind of like now we-- you use the full model to generate in 100 steps, and then you take a model that only generate 10 steps and let that model to learn from the perfect one. Ethan 00:38:25 : why this work Swyx 00:38:27 : Strong to weak seemingly. Ethan 00:38:28 : It is. It’s kind of Swyx 00:38:29 : Distillation Ethan 00:38:29 : kind of like strong to weak. the-- from the modeling perspective, the strong model, the teacher model is trying to model the image and videos of inter-internet, and that distribution is extremely complex. But the step distilled model is just trying to learn from the teacher. The teacher is a model, and the size is fixed, as the distribution is much simpler than the whole internet. That’s the intuition I have why step distillation can work. So usually these models serve in productions, they only run in a few steps. In Cosmos, I believe we have, we have like four step and eight steps. If you do some simpler task, image-image translation, it can even run in fewer step, like one step in Cosmos Transfer. Swyx 00:39:22 : I think this is the same intuition that guides a lot of the consistency model work. I sent you a link for, SCM. I don’t know if you covered that. To me, that was actually one of, the most impressive papers I’ve ever seen from OpenAI. Swyx 00:39:34 : That this is the unifying grand concept of consistency models. I don’t know if you have any comments on this. Ethan 00:39:41 : So there are, there are a few different approaches, Swyx 00:39:46 : Oh, yeah. Here it is. Swyx 00:39:47 : Two steps versus twenty or 100 steps, whatever. It’s already done. Ethan 00:39:52 : So there are, there are a few different approaches, for example, consistency model, and there are also Actually, we shouldn’t forget GAN. So GAN, actually, that was, that was the OG of Swyx 00:40:05 : OG Ethan 00:40:05 : step distillation ‘cause it trained just one step to begin with. So actually, a lot of, uh-- For example, there’s a distribution matching distillation which use, which uses GAN, as one of the laws for distillation. It-- GAN just tells you, “Hey, generate an image,” and then Ethan 00:40:31 : it has a discriminator to tell, is this image real or not? So the model, the model just need to learn one of the distribution, not the full distribution. Because in training, the model is asked to reconstruct the ground truth image from the internet, which is extremely hard. And in-- When you’re training GAN, it’s a step process. It’s just a, “Hey, you generate image. Does this image look as real as the image from the internet?” Which is a much simpler task. And, yeah, combining a lot of these approaches together, people typically do that, like consistency model and distribution matching and GAN, and we can get these few step models. Audio-Video Generation and Time Alignment Swyx 00:41:21 : Then there’s one step I wanted to add, which is audio and video. Ethan 00:41:26 : So, Grok Imagine zero point nine, I believe it’s, it’s a first audio video transmodel deployed at a large scale. So Swyx 00:41:39 : And that was your first model? Ethan 00:41:40 : that was, Grok Imagine’s first model. It’s, it’s audio video, joint generation. I think the hard part is, the modality alignment, ‘cause before this transmodel, we have, we have text to video alignment. We have this, correspondence between text and video. Typically, most of the VLMs, they understand images and videos. Video’s very rare, and they don’t understand audio mostly. And if you look at the audio generation on the LLM side, you can talk to them perfectly fine, but if you ask them to sing a song or something, it typically is not very good. Also, they don’t have, they don’t have music either. The hard part is thatUh, actually audio has two component. It has like a discrete component, a continuous component. The discrete component is like the language. Ethan 00:42:44 : So when we speak, it’s just, some Swyx 00:42:47 : It’s an ASR issue, yeah. Ethan 00:42:49 : It’s, it’s text token with some characteristics, I would say. Ethan 00:42:54 : But music Swyx 00:42:56 : I think the speech guys would disagree with this. Swyx 00:42:57 : Like disfluencies and then, Vibhu 00:43:00 : There’s tones you can get angry. Ethan 00:43:01 : Well, I say largely. Ethan 00:43:03 : the mu- but the music is completely different. It’s, it’s very continuous, and you cannot model them like discrete tokens in language models. this is like the hard part for models is, not to mention we have to align text, video, and audio together. Ethan 00:43:26 : So Vibhu 00:43:26 : How? Ethan 00:43:28 : So significant-- some significant challenges are like-- So first, like we talk about as the VLMs, they cannot understand most of them cannot understand audio. Ethan 00:43:39 : So you have to have some way to do the synthetic data generation for audio. You have to caption the model, and that involve, that involve synthetic data and human data effort a lot. And not just surprisingly, most of the LLMs are very bad at recognizing, like the beat, tone, and the details of the of music. They can, they can give some general prediction of which song is this, but it’s very hard to describe the details of the music. like we mentioned in image generation, like you have to describe image as detailed as possible so that someone blind can reconstruct that. So here is like someone Vibhu 00:44:32 : Deaf Ethan 00:44:32 : someone deaf can reconstruct how the music sounds like without actually listening to it. Maybe you can think of it need to have the-- or they call the script. Vibhu 00:44:49 : Subtitles, yeah. Ethan 00:44:49 : You gotta have all the details of the music, and the dialogue. Vibhu 00:44:55 : So is the challenge there typically stuff like music and audio, or is it just Like is there a baseline? Okay, there’s enough data where we can understand, narration, conversation, but there’s nuances in audio that’s where you hit all the data issues or is it just from stage zero, you just do it all right? Ethan 00:45:15 : So one important thing is like the alignment. So the model, the model has to know like the video and audio, the, uh-- it has to have a time-based alignment, like at which time step the video and the audio token correspond to each other. But we actually don’t have this kind of alignment for most of the other modalities. If you think about like text and image, text and video, they are loosely aligned. So you can, you can have a description of what’s going on in the video, but you don’t have to exactly, You typically don’t have exact description, oh, at, time step one second like what happened? Vibhu 00:46:02 : It’s very Ethan 00:46:03 : At time step two second what happened Vibhu 00:46:03 : coarse. Yeah. Swyx 00:46:05 : So what was the ideal time step? You have to oblate it, and then it’s like four seconds or something. Ethan 00:46:09 : So that comes down to how you design the model to, for the model to be aware of as a time, as a time modality. So the model is like a time aware. And that’s something pretty unique if you think about LLMs. So if you ask LLM to complete a task, say they, uh-- you ask them and they will say, “Oh, this task will probably take twelve hours to complete,” and they come back in one hour. Say “I’ve already spent two days on this and I’ve exhausted everything.” Ethan 00:46:47 : So the LLMs them-themselves, they don’t have a sense of time there. Vibhu 00:46:53 : I actually don’t think that’s just them not having a sense of time. I think it’s somewhat based, right? Vibhu 00:46:58 : Like you tell someone, “Okay, go work on this feature. Go implement this,” there’s a general understanding you would have of how long that would take without LLMs working at LLM speed, right? So you think back like two years ago, if I tell you to like build me like a new front end for latent space, have a search bar, have all this, you’ll estimate that it’ll take a few days, right? Vibhu 00:47:19 : So you tell an LLM, “Go build this.” It’ll take me a few days. But I think it’s somewhat grounded as opposed to them not having the best-- Not saying that they have a great understanding, but I think that example is like you can see where it comes from, right? You’re trained on all over the text. Swyx 00:47:35 : They’re, they’re trying to estimate what a human would say. Vibhu 00:47:37 : because that’s what the, that’s what the data kind of represents. It’s not them Ethan 00:47:41 : It came from the corpus on the internet. People have a estimate of how much time. Vibhu 00:47:45 : And not even just in direct like training samples, right? Just your world understanding of tokens of how long stuff takes, right? Go read a book. It’ll take you a while, right? Vibhu 00:47:56 : Even if you do nothing but read a book, it takes a few days. So yeah, LLM, I read it took me a few hours. Vibhu 00:48:01 : It’ll take me a few hours to go through this research. But this is a tangent. Swyx 00:48:05 : Somewhat, yeah. Swyx 00:48:06 : This is a train of thought I haven’t really expressed until now is, which is basically like a full world model must also be recursive, meaning that the participant in the world model must also be aware that they have a world model. which is like this whole recursive thing down the, down the line. but yes, and that the world model can be wrong and that they need to update it and blah. Yeah. We’ve, argued this on the, newsletter as well, that there needs to be sort of recursive or adversarial world models. World Models: Real-Time, Long-Horizon, Interactive Video Vibhu 00:48:34 : just, to ask, how do you define world model? Swyx 00:48:38 : Oh, yeah, let’s go there. Ethan 00:48:40 : So Vibhu 00:48:40 : So just for context, we talked about, video generation, and then there’s a-- if you say there’s a distinction between world models, what’s your, what’s your definition? How do you see the two? Ethan 00:48:53 : So disclaimer, I’m not going to debate, what is world model. Yeah. there are many definitions, so I’ll just talk about my definition. Since I came from the multi-model, multi-model domain, so mainly talking from video. So world model is like real-time interactive long horizon videos. So there are three parts. so we-- let’s talk about them one by one. So the so interaction, so we just, we just look at Facebook and neural computer. So the interaction part of it, so you, world model can allow you to interact with them through keyboard, mouse, and maybe also voice. So these all is-- all is a modality. You can, you can interact with the model, and the model should respond reasonably. Second part is real time. So once you, once, say, you move your mouse, if, say, the world model generate a game, how fast can the game respond? So if you’re like professional CS: GO players- -my say, oh, you have to respond- He’s beginner within sub ten milliseconds or- Yeah even less. So that’s not most of the- No, sixty FPS. Let’s go. Oh, three hundred FPS. Oh, five hundred FPS. Wait. okay, yeah. I didn’t do the math, but yeah, okay. Uh- Yeah, three hundred FPS, that’s a three millisecond. So you have to respond- Oh, shit. Okay. Yeah Ethan 00:50:29 : within a millisecond. Most of the video models cannot do that. Yeah. And, but if you, say, if you have a video model that is, say, like a digital human, the response time might be more generous. Maybe typically, for real-time voice interaction, it’s like two hundred millisecond. So that’s, that’s much more generous. But even two hundred millisecond is pretty, it is pretty tricky, ‘cause remember we mentioned Ethan 00:51:01 : you have this, temporal compression coming from the VAE. So if you, if you don’t compress the temporal dimension, your sequence length is going to explode. So if you want to have this real-time, real-timeness in your model, you have to do is one context problem. And the third part is long horizon, ‘cause we-- if you’re not going to just play with, video games just, a few seconds, most video models only a few seconds. We’re going to play with minutes, hours. The model have to be able to generate long-form content. Ethan 00:51:42 : So putting these three together, it’s, real-time, long horizon interactive videos. I think the final state will be, for example, like a video, a video version of Playbook, where you can, you can interact with, a neural computer. You move your mouse, and you click on the generative interface, and it will reply to you through pixels- generating in real time. But getting there, it’s, it’s a very long way to get there. So one of the first step, at Grok Imagine, where I led a small world model team there, was to build video extension. So, video extension- it’s the first step of interactivity. Yeah. It’s, it’s the first step. Yeah. So it’s the first step- You have it here, video editing, yeah. Yeah. Yeah. So the first step is because, this unlocks long horizon videos. Typically, for most of the video generation models, you give it a prompt or an image as an initial frame. You generate video, that’s it. That’s just, one time, done. And some creators would try to, use the last frame as a first frame for the second video. It can-- sometimes it works, but if you do it a few times, it says the quality would decrease. And- It doesn’t have that context- Yeah over the full video, so the temporal- Yeah, exactly. Yeah, ‘cause you only gave it the last frame, of course, right? Yeah. Exactly. And- it’s actually a pretty fun hack. if you’ve seen like- Oh, no, he’s saying something better. Yeah. And for example, like Vue, I remember Vue 3 has like a second context of the last video. It is slightly better than using the last frame, but it has the same problem-- similar problem that it, the quality would decrease. if you extend a few times to, one minute, the video quality would look much worse than the first video. Second, another problem is that the model doesn’t have long-range knowledge of, what’s happening before. Say, if they generate some dialogue, some, two people speaking, and their voice might change, over some time, especially if the second conditioning, it does not cover the previous context. So these are the core challenges. So the Grok Imagine video extension, it has historical context of all of the previous generated videos. It can, It has, it has the context of, who is speaking and what objects have appeared and everything, having that to generate the next video. So if we naively do this, you can imagine, just, put all of the previous history video tokens into the context. The context lens will easily explode. Especially for video models, that can be like a few, a few million context, I would imagine- context lens. Yes.Yeah. Swyx 00:54:58 : Let’s run with that. Ethan 00:54:59 : for example, like in Cosmos, I think just five seconds of video is like a fifty K or sixty K number of tokens. So like if you do, if you do fifty second, that’s a five hundred K tokens. If you do longer than that, easily explode. This long horizon, problem was the first step we’re trying to solve world model. It turns out people, yeah, people love video extension. Like a lot, a lot of the creators love using video extension to create longer form videos. This is the part I liked that you have a, you have an intermediate step toward the final goal instead of just a straight shot to the final version very much. Swyx 00:55:48 : But I can see you have a strong vision of where we want to end up. Long Context, Redundancy, and Efficient Interactive Video Vibhu 00:55:51 : Does it seem like it’s an efficiency issue? okay, we’re at a few million tokens context,. If you draw the parallel to language models, we had very short context, two thousand, eight thousand, then, you scale it up one million, ten million. sure, there’s effective context, but at the end of the day, it’s just what’s it worth? sure, there’s a whole training data side. In video, it might be slightly easier ‘cause we have a hundred million token video, right? Just take a movie with the full context there. Like is this efficiency from an inference standpoint that like it’s expensive, but we know how to solve it? Or like why is this not the approach? So like my broader point was on your second point of world models, you say it needs to be interactive and live, right? You should be able to play a game and see the interaction live. So one thing I see with research is a lot of what you actually serve is different than what you build, right? So we talked about distillation. You train big model, you distill it, you do quantization, speculative decoding. We do all this stuff to serve it efficiently. Should we not just have a solution, like a world model that can interact well, do inference optimization, serve it, distill it secondary, so make it real time after you solve it? So like a-- another parallel is say, continual learning, right? What we need is someone to solve it and show it works inefficiently. Give it a few years, people will make it efficient. Same thing with regular attention, right? It worked. Over a few years, people have different forms of attention, and we’ve scaled it to be efficient at log context,? So kind of two things there, right? One is it seems like it works. You’ve scaled it. Can we not just scale it a lot more efficiently over time? Do we need a separate approach if this works? And same thing with interaction, right? if we can get it done, like if we can solve some way that it works, we can solve making it more efficient from an inference standpoint later. Ethan 00:57:53 : that’s actually a very good point. So in videos, there’s actually a lot of redundancies. So we solve a lot of the pixel redundancy from VE, but there’s more redundancy in long range and long horizon videos. Say, if a character appear in the first clip and then it disappeared, it only reappear at the end of the video, you probably don’t need the-- the context, like in the middle of the generation. So you only need that character, where you need. So that’s why, I helped build another feature. It’s a reference video. Vibhu 00:58:36 : Is it here? Swyx 00:58:36 : is it the same model release or different one? Ethan 00:58:39 : It’s a different one. Ethan 00:58:41 : You probably need to search on Swyx 00:58:43 : I’ll find it Ethan 00:58:43 : X reference to video. Ethan 00:58:46 : So reference video allow you to like upload up to seven images as condition and generate the video. Say, if like I want-- it can, it can be characters or objects or even scenes. Say like I want, I want condition on, Sean’s selfie and holding a blade Swyx 00:59:07 : We have a dog Ethan 00:59:08 : or whatever. Swyx 00:59:08 : We put the dog in the thing. Ethan 00:59:09 : you can put them there and the video models will generate the video from and copies the context over. So that can solve a lot of the problems there, like the long context problem. It doesn’t need to have a very long context, but it’s-- I feel like it’s an intermediate solution. The model Swyx 00:59:29 : It’s cheating. Ethan 00:59:30 : the model should be able to like selectively know, where should I draw the references. So say if I want to generate a movie, I generate it autoregressive, like a ten second at a time or something. And now this character appear, I can look back to where it first appear and, bring that back. Yeah, this one, I put the references. Yeah, that’s, Optimus, Einstein myself, Annie. Vibhu 01:00:02 : Oddly enough, I used Grok Search to find it, and it pulled your LinkedIn post. But yeah we found it. Ethan 01:00:08 : Interesting. Vibhu 01:00:10 : But xAI’s Underrated Work, Culture, and Watermarking Swyx 01:00:11 : this is a problem. This is not your fault, but like XAI doesn’t communicate all this work that you do very well because they just have the model release and then that’s it. But actually, these details are very good. Swyx 01:00:22 : As far as I understand, everything you just described is state-art, like no one else has done it. Vibhu 01:00:30 : A lot of-- yeah, I have a lot more Swyx 01:00:32 : And then, and then you just put this blog post with the cookies. I’m this is not enough,? Swyx 01:00:37 : but I, obviously this is like the high level numbers that people want to know. But no, okay, so Vibhu 01:00:42 : And I wonder, like part of that is also some labs don’t share research into what happens. And if Swyx 01:00:50 : No, but this is literally bragging about how good they are, right? Swyx 01:00:54 : Like, why would you not say that you are capable of extending with full context? this is not a secret sauce. This is like we did the work. yeah, I don’t know. Ethan 01:01:02 : different labs have slightly different communication styles. Swyx 01:01:07 : Anyway, if anyone from XAI is listening we are always happy to help you tell your story. Yeah, okay, so you did references, and I think, I think kind of the point you’re, you’re making is it is sort of like a kludge, right? this is-- you can do seven, but what about 100? Swyx 01:01:23 : Right? Then you need a completely different thing. Ethan 01:01:26 : So I think it’s-- this is, a mechanism to, select the context from the history, and you might not put the entire history into the context. for example, there’s a paper called Frame Pack, which have Ethan 01:01:41 : a heuristic that the latest history, the last one second, I put the entire history, and the history before that, I would, compress it and makes the video smaller. So they follow this pattern, this build overall pattern that the maximum sequence length is fixed. So the further you are from the current frame, you have a smaller image. So this is just a heuristic. I think it can be more automatic. The model is aware like which history part of it can be select. So this part of the research is actually being actively, worked on by a lot of people. It’s also quite interesting. I feel this is actually, this part of long context is a little bit ahead of the LLM part. Ethan 01:02:31 : So for example, like in LLMs, if you-- so contexts keep growing. Let’s say if you call tool and the tool call history is extremely long, that’s still in context, and keep growing, keep growing. Even if you switch the topic to something else, the whole context was there. There are some agentic harnesses that help you to, say, prune the tool results and, prune Like when you, when you query a file, only show like the top 200 lines or something. Those were very heuristic-driven. Swyx 01:03:08 : For listeners, we did a write-up on the cloud code, leak where there are eight different kinds of pruning, including like you prune the tool results and all that. So you can, you can read up on that kind of thing. Ethan 01:03:17 : I think, one breakthrough in continual learning might be like a way to automatically, manage its own context. Swyx 01:03:27 : These are all heuristics, and they will be replaced by machine learning. Ethan 01:03:30 : Interestingly Vibhu 01:03:32 : The Ethan 01:03:32 : the same thing is being researched in both LLMs and video models. Vibhu 01:03:36 : The interesting thing is also like in the paper you showed, it’s actually happening at the model level, right? Compared to like language models, sure, we have base attention, but we’ll do our own compression, we’ll do our own pruning, which is separate from model error. Vibhu 01:03:49 : Eventually, it all just boils in, hopefully. Swyx 01:03:52 : I think this is a form of like attention, but like also know sort of reasoning attention. I feel like that’s different than normal attention. Swyx 01:04:03 : Does that, does that make sense? Ethan 01:04:04 : It’s, it’s different in the sense that attention, not to mention, set sparse attention aside, like normal attention Swyx 01:04:13 : Like UKV, yeah Ethan 01:04:14 : you have to attend to all of the tokens. Ethan 01:04:17 : So you don’t have a high-level mechanism to drop which tokens do-- you don’t want to attend to. As humans’ attention span is surprisingly small. Ethan 01:04:28 : You can only remember 11 digit of a phone number. Swyx 01:04:32 : But I have feature detection, right? I can detect, oh, that’s a sequence of one, two, three, four in a phone number that is 11 digit. Vibhu 01:04:39 : Very good pattern matchers. Ethan 01:04:41 : But humans’ context can-- like attention can work because we can dynamically pull in, context from different places. The same mechanism, I think is going to happen for LLMs and video models. I think we have Swyx 01:04:57 : RLMs is recent-- is on, it’s on the recent work is there, which is not that, crazy, but it’s just recursive. Vibhu 01:05:04 : I think it’s somewhat inherent in models too, right? Like you Swyx 01:05:06 : No, here’s a nice example here Vibhu 01:05:07 : you pull up these, you can read it fine, but, language models are also very good at slop parsing. you have a trans Swyx 01:05:15 : I throw my typos in there, it doesn’t matter. Vibhu 01:05:17 : You have a, you have a transcript, you have whatever, just throw it in and it’s very good at parsing through noise. m-- that may be a brute force. It can look over a reason over it, but there’s, there’s parallels to both. Swyx 01:05:31 : I think it’s just really fascinating how you relate the world models stuff to the video generation, which I don’t think a lot of people hear directly, from people like you. So I think that’s really helpful. Any other work? Do we cover like video, audio, world models, any other stuff in that omni Swyx 01:05:48 : team,? Vibhu 01:05:49 : Or any other work at XAI you want to talk about? Seems like everything we see publicly announced, “Oh, cool, cookies.” And then there’s so much more to it. Swyx 01:05:58 : There’s a lot of depth. Vibhu 01:05:59 : Any underrated stuff, just at the time there? Ethan 01:06:03 : I feel the, as a culture, it is quite interesting and a bit underrated. So the culture is, the culture is three sentences: move fast, build No goal is too ambitious, and the first principle. Like early, the goal set was very ambitious. It wasn’t very-- this wasn’t-- it wasn’t possible to achieve when I, when I was thinking, first thinking about it. Like for example, like build something in three months. And Vibhu 01:06:36 : Was that “Okay, we’re starting team, we want image, we want video. Do it by this deadline.” Or, how do you work back? Like was it just, “Okay, we have a rough by, this date we want something out,” or is this like Ethan 01:06:52 : That’s a very good point. So it’s from first principle thinking. Ethan 01:06:56 : If you think about, people might say that first principle thinking applied more to the physical world than the models. I would say, for example, like if you think about-Some limitation, for example, acquiring data, like how fast can we acquire the videos? And if you think about training the models, what’s the iteration speed for training a model end? And how would adding more GPUs accelerate that timeline? And maybe if you need human data, like what’s the turnaround time for human data to arrive? If you put all of those together, that is first principle thinking where, oh, like what is the timeline? What’s the minimum number of days that is possible to achieve something? Swyx 01:07:52 : I think there’s a-- this is a lot of Elon’s type of thinking, right? He’s like-- I think he’s famous for saying that the only law you can’t break is the laws of physics, something like that. Swyx 01:08:01 : Just broadly, you worked a lot with Elon. Ethan 01:08:04 : I, one benefit is working at xAI, you got a chance to interact more with Elon. So I was very fortunate to get a few retweets from him, and that was quite fun. And, he also worked very closely, with people. like people imagine online, like he’s very hands-on. Vibhu 01:08:34 : There are two things. one-- So I was actually looking up, Elon retweeting you. I’ll pull it up. he talked about you tweeting that you have a really good voice mode. I don’t know Ethan 01:08:47 : Oh, me? Vibhu 01:08:47 : No. Him. Swyx 01:08:48 : Oh, I also did it. But anyway. Vibhu 01:08:49 : I actually-- So I would DM you feedback on voice mode because I was “Wow, really good.” And then I’m “Ugh, this sucks.” But, I don’t know. Anything you want to talk about your voice mode, building it? Was it a team you worked on as well? Ethan 01:09:02 : Oh, that’s actually not part of the team I worked on. Swyx 01:09:05 : He probably worked on more of the video. No, but Grok Voice actually Vibhu 01:09:11 : Grok Voice Swyx 01:09:11 : like very good. I-- This is one of those things where first of all, you can speak at 2X, which is fun. Swyx 01:09:16 : which I listen to 2X, so I like to speak at 2X. But also I think like the interruption was better than Gemini. I don’t know how it compares to ChatGPT real time now, but as far as like driving was concerned, like having Grok in my Tesla and like driving, I think it was like-- it’s a really good experience. Vibhu 01:09:34 : He likes voice mode. But also, just the crazy reach by Elon Swyx 01:09:40 : Fifty million views for just saying, “Yes, true.” Vibhu 01:09:43 : That’s true. Swyx 01:09:44 : Oh my God Vibhu 01:09:45 : but, it’s, it’s pretty cool how fast it came out. the other thing is the safety aspect of video mode. Anything interesting to talk about there? So Swyx 01:09:56 : spicy Vibhu 01:09:57 : spicy question. Ethan 01:09:58 : A lot of the countries where they don’t allow like a generative data-- generative AI videos without watermarks. So in all of the-- those countries, Grok Imagine had watermarks, and a lot of the-- a lot of the takedowns of the videos were also happening extremely fast. Swyx 01:10:22 : it’s, it’s part of running a social platform but also it transfers nicely to the GenAI side. Do you have a perspective on SynthID versus other kinds of watermarking? Ethan 01:10:33 : it’s going to be Ethan 01:10:37 : it’s going to be harder and harder to detect, the Yeah, these things. So SynthID, one thing is, previously it was only Google, and now, like a lot of different labs Swyx 01:10:52 : OpenAI adopted it Ethan 01:10:52 : are also adapting it. Ethan 01:10:54 : As-- A limitation is like the technology The paper was out there, and people can reverse engineer like how to get rid of it. Ethan 01:11:05 : And it’s-- I think even as it advance, it’s, it’s still possible to reverse engineer it. Swyx 01:11:13 : so if you are interested, you can go onto Reddit and people have taken out the exact I don’t know, what do you call it? Mask or pattern that Google applies, and then you can apply it onto any Google-generated photo, and you can reverse out the SynthID. Ethan 01:11:30 : And it’s, it’s also harder and harder to just judge by eyes. I remember like a couple years ago, there was like six fingers or something. It’s very obvious. Vibhu 01:11:42 : My current is actually the audio. I feel like the audio is really lacking. my way to tell if something is generated, outside of okay, I think I’ve seen enough, I have a decent eye, the audio matchup, especially of Sora, is not great. It’s all similar style. But there’s Swyx 01:11:57 : I see. those are minor imperfections. Swyx 01:11:59 : I think the point is that like-- Actually, my closest reference to this is also Ian Goodfellow, ‘cause I think he did like the adversarial GAN thing where like it’s okay, here’s a picture of a zebra. Then you like change one pixel, and it becomes a panda. Swyx 01:12:12 : Right? This is like-- this is like a classic computer vision issue. Ethan 01:12:15 : If you think about how these models were trained, like I, like I mentioned before, like GAN was in the training process. The objective of GAN is you-- is the model generates an image, and the model, there’s a judge to tell like if the image is real or not. The model is trained to make the image more real. So as the model become more and more advanced, it’s going to be harder and harder. For me personally, now I have to judge by Ethan 01:12:49 : if the-- these videos have logical sense. Ethan 01:12:53 : If these, this video Swyx 01:12:55 : Have a world model. Swyx 01:12:57 : No, I also like it-- the audio is too nice, like too studio quality. The lighting is too good. The skin is too clear. the-- basically, the lack of imperfections. Vibhu 01:13:10 : Do we have a good way to do reasoning in diffusion? Like is that what separates video generators from world models or in, -We really know how to apply it to other regressive language models. Is there a parallel for diffusion video gen world models like on that point, right? Is Swyx 01:13:30 : He has a thing on video agents. Ethan 01:13:31 : that’s a good question. Yeah, actually, I have a, I have a pretty big claim. The intelli- the visual intelligence are actually mostly coming from language. these video models, especially from now, since the diffusion model technology is more mature, the every time you see there is some improvement on these models, I would say mostly, this, again, comes from language model, not coming from the vid- the video model itself, like the video distribution models themselves. In Cosmos, that could be Typically these models, they have two parts. there’s a, there’s a prompt rewriter or the prompt up sampler part. I think in Cosmos, we use Llama or we use Mix- Mixtro. And the Cosmos video model itself is only 7B, and the model, the language model Prompt Rewriting, Video Agents, and Agentic Generation Ethan 01:14:35 : is a prompt rewriter. It’s, it’s bigger than that. So the prompt rewriter’s task is to take user instruction and convert it to extremely detailed description of the video. So because the video, the visual-- the video distribution models, I would describe, they’re kinda dumb because they take the input Ethan 01:15:03 : instruction literally. Because in the training process, remember that we have to describe the video as detailed as possible when we’re creating the synthetic, text pair. So this model, they take those kind of instruction to generate the videos. So in-- when you’re taking the user instructions, the user instruction usually are simple. Just say a cat or something. If you put a cat in the video model, they would take that instruction literally. They would literally show a cat, a cat in maybe a white background because you didn’t describe the background. The cat is not moving because you didn’t describe it. It takes the instruction quite literally. It’s kinda, it’s kinda dumb. The prompt rewriter is actually a much bigger model. It’s a language model that takes, the user instruction and expand it. So the thinking process you mentioned, is from there. So if you, if you look at like GPT image, like you generate a image in three minutes. Three minute is not all like a pixel generation. A lot of time is spending Vibhu 01:16:19 : Prompt writing Ethan 01:16:19 : on thinking. Ethan 01:16:20 : So prompt rewriting now have evolved to, not only just as thinking, it can, it can also be a agent, a agentic model. For example, say you want, you wanted to generate the image of today’s news. So the-- So it’s likely they’ll go to fetch today’s news online and then, process and digest them, then organize the layout and generate it. Another thing quite interesting is, Vibhu 01:16:53 : If I’m not mistaken, these are-- it’s no longer a diffusion model though, right? It’s autoregressively Or is there still Ethan 01:17:02 : There are different approaches. For example, Gemini Omni. Since they said it’s Omni, I believe it’s a, it’s a single model. Maybe it’s something it’s a language model with a diffusion head or something. Like the language model do the thinking, do the agentic tool calling, and then it would, use the diffusion head to generate the image in the end. There were also approaches like Cosmos, where you have a separate language model and separate diffusion models. And there were also like a purely language model, like you discretize the images, and then you generate the image as discrete tokens. So there are different approaches. I would say like Vibhu 01:17:44 : One of, one of the claims I’ve seen for why these approaches struggle is because a lot of the benefits for how we currently learn reasoning with language models is you basically iteratively generate reason. You have your thought, and then you work on that answer, right? So if you have like Omni model and then diffusion head, you can’t feed that back in to continue reasoning, right? So you can’t go like text, image, text, image. You can’t reason on the output and then go back to diffusion. But in the new Gemini Omni, you would be able to, as long as you have diffusion. Ethan 01:18:15 : I’m not sure if Vibhu 01:18:16 : But Ethan 01:18:16 : they have that process. it’s definitely possible in the Omni paradigm. Ethan 01:18:22 : So if you think about like traditional multi-model language model, they would have a VIT encoder that can encode the image. So if they have a diffusion head, they can generate the image and then put that back into the VIT encoder, encode that, and then do the iterative refinement if the result Yeah. Swyx 01:18:44 : I think you have to jointly train the VIT and the diffusion to make that somewhat reasonable, ‘cause otherwise you’re kind of mismatching or feeding in slop. Vibhu 01:18:55 : I think it depends on the stage of training. You might be able to freeze it. But anyway, also just on your earlier Swyx 01:19:00 : Wait. I wanted to also make explicit. We do know that NanoBanana and GPT image are autoregressive, language model with diffusion head. Swyx 01:19:09 : as far as I can tell from your description of Grok image, it is not. It is, it is end. Ethan 01:19:14 : I cannot Swyx 01:19:15 : You cannot Ethan 01:19:15 : comment on that. Swyx 01:19:16 : Well, the way that you described it. but, yeah, I think it-- there’s, there’s different approaches, right? Like you started off saying prompt rewriter is, the-- a big part of the intelligence. Vibhu 01:19:24 : and even on that, I think everyone should try using an early diffusion model. If you’ve used Stable Diffusion one or whatever, if you’ve seen the prompts ultra-high res, four K this style, oh my God, the first time I tried one, you don’t talk to them like language models, right? Your prompting is very, comma separated Swyx 01:19:43 : It’s literally talking in the labels that were in the data set, right? Swyx 01:19:46 : But basically, I’m just trying to make the point that prompt writer and then image is different from autoregressive language model with diffusion hit. Right? They’re different things. Ethan 01:19:56 : they’re different. Swyx 01:19:57 : Just wanted to establish. Ethan 01:19:59 : I’d say, the common part is, the image part. So it’s, it’s quite surprising that, a lot of the improvement came from the Swyx 01:20:12 : Language side Ethan 01:20:12 : the thinking the tool calling. So I still remember, in Cosmos, I generated a happy sheep and can if without any rewriting, it’s-- it looks so, CGI, and after rewrite it looks, it looks so beautiful. Ethan 01:20:31 : I think Swyx 01:20:32 : Without any joint training. Ethan 01:20:34 : actually, without any joint training. it’s-- with rewriting, it’s already much better. See, a very interesting thing, what happened is the video agents, mostly language models, will call these, generative model, either it’s a separate model or a diffusion head or whatever, as tool. So this model can iteratively refine the results or even, generate longer content through a very long train of thought. It’s actually very similar to how human create art. So we don’t, we don’t generate the pixels directly. We literally draw something on And I think through this process, the-- these models not only use diffusion as one of the tool, it can also use traditional tool. It can also use, image editing tools from Photoshop. It can use, video editor, FFmpeg, whatever, to take combination of these and the generative AI technology as a, as a set of tool, and they can, they can iteratively create a better, a much better, video for production-grade quality. If you look at existing, professional creators, they don’t, they don’t end at, generating a video from these models. They would take this video to their editor and edit here and there. Swyx 01:22:11 : So much post-production in And sometimes actually, the reason the video is good is not really the video model, it’s actually the editing. Swyx 01:22:21 : And yes, we also are engaged in the same process as well. Would you love to use a video editing model? Ethan 01:22:27 : Actually, there’s, Grok Imagine Agent beta. That was the, that was the first attempt in that direction. Ethan 01:22:38 : So I think, the process would be similar to like Vibhu 01:22:44 : It’s just agent mode. Ethan 01:22:46 : you can, you can ask it to Swyx 01:22:48 : There’s no blog post for it Ethan 01:22:49 : maybe generate a minute, video, which is not possible if you ask the same prompt to video models. But this model will ca- literally call different tools to do that. Ethan 01:23:05 : So yeah, this is actually an interesting thing. So when we first released, a video editing model, I see on X some people try the video editing feature with, “Edit this video to be one minute.” ‘cause they didn’t understand how video editing work. Video editing typically is just a removal, add, replace, style transfer, this kind of thing. But that’s actually a valid request under the assumption of video agents. So these agents should be able to understand these kind of, long horizon tasks to be able to easily, create a long-form video. I think this is, this is really fascinating ‘cause it’s kinda take-- it’s taking the same direction as first you have these, assisted-- assisted coding, kind of like tab completion, GitHub Copilot. And from there, you gradually evolve to Codex and Cloud Code, where you do things fully automated. So in agent, in Grok Imagine Agent mode, you can, you can still go in there and do stuff by yourself. Ethan 01:24:22 : gradually, as the model capability increase, it will be able to do everything fully automated. Swyx 01:24:30 : I like that. okay. Ethan 01:24:32 : That’s good. Swyx 01:24:32 : So it looks like it’s still generating. Vibhu 01:24:34 : Also, I did notice the Grok image gen was always very fast. I don’t know if this is something you guys benchmarked, but, this is just a tangent. Compared to what I used to use before the latest OpenAI’s image gen, and same with Gemini Nano Banana, I would oftentimes use Grok just for the speed. Swyx 01:24:54 : It’s, it’s in the benchmark somewhere that’s Vibhu 01:24:56 : It’s Swyx 01:24:56 : in the Imagine API blog post that they have all the speed things. Swyx 01:25:00 : it mostly combination of distillation plus inference. Ethan 01:25:04 : There are a bunch of things. we talk about distillation, and if you talk about thinking, if you don’t have any thinking budget, the model can just think three minutes and then come back to you. And also, inferenceThe inference infra team was very talented, and they were, they were able to accelerate a hell lot of these models. Swyx 01:25:27 : my comment on the, on the video agents things, I’m trying to figure out, when people say video agents, when you initially told me about your bet on video agents or your vision for video agents, I was a little bit disappointed. I was “you mean, like models are tapped out, now we have to do agents?” But, I think you have to, right? The question now is, how much model training is it really going to make a difference versus just building a better harness? Like you said the models don’t have to be jointly trained. you can just take an shelf frontier reasoning model, slap it on a harness, give it Grok as a tool. That’s it. That’s your video agent. Doesn’t seem super satisfying. Obviously, you can train and get some more percentage points of per- performance. But, if your central claim that the majority of video or generative media, alpha or whatever, is actually coming from language intelligence and not, image diffusion or video diffusion, then that is the future. Vibhu 01:26:30 : it’s pretty cool Swyx 01:26:31 : It’s just like primarily just weight. Vibhu 01:26:33 : If you pop back at the example, it generated frames. Sorry to interrupt, it’s been saying “Okay, I’m gonna start stitching these frames together.” Swyx 01:26:42 : So Vibhu 01:26:42 : It’s using FFmpeg like using code. Swyx 01:26:43 : This is what GPT Image Pro as well is doing, right? Swyx 01:26:46 : Like, this is also just writing code in the background and then just Vibhu 01:26:48 : Stitching Swyx 01:26:49 : doing an image pass on the final output. It feels dissatisfying for the people who want to just train models. Vibhu 01:26:54 : It’s interesting, right? it’s, it’s also somewhat exciting. Like you brought up earlier, a lot of the gains don’t come as much from the video. I think you can see that in the language model space too, right? Anthropic, very good at coding. They’re multimodal, not the best, right? They have basic input PDF, but there’s clearly a disconnect in the quality of their image video processing, audio processing, yet intelligence very top tier. Other labs, Gemini, OpenAI, xAI, you can add modalities, but it’s not like they’re unlocking crazy capabilities, right? So it’s interesting. Ethan 01:27:32 : It’s interesting to see that, like the video model’s capability increase actually come from language model being more intelligent. I think video agent, like it can unlock more stuff than my- you might imagine. So there’s a few things. So one thing is when we are prompting these models, so most of the people were actually not very good at prompting. Ethan 01:27:59 : Actually, language models have a better sense of how to prompt AI models. AI models know AI models better. So if you jointly train these models, maybe the model have a better sense of, how to prompt each model. Like a different model Vibhu 01:28:15 : Of course Ethan 01:28:15 : might be different. Another thing is it might not as simple as just, like generate a few clips and slap them together using FFmpeg. Like you might-- there might be more like image and video editing tool appear in this process. Say, if you want to exactly add a blob of text at this timestamp, the videos model-- video models might not get that intention very precisely. Ethan 01:28:48 : But these are possible using these deterministic tools. The long-- The video agents can use all sorts of tools, so you don’t have to put all of the capabilities into the generation model itself. Swyx 01:29:04 : I think that’s very true. no, so for what it’s worth, I think you’re right. I think that this will be a big category. I think probably you are predicting like the next one year in video is gonna be all this. Vibhu 01:29:18 : Do you have a time prediction for how-- when this stuff ramps up? Like Swyx 01:29:22 : they already started. Vibhu 01:29:23 : Is, Swyx 01:29:24 : It’s not very good yet. Vibhu 01:29:25 : Are we so-- No, it’s so, it’s so good. I think the last one’s just longer. Vibhu 01:29:29 : it didn’t give me a minute. Ethan 01:29:30 : Last thirty-six. Vibhu 01:29:30 : It gave me thirty-six seconds. But are we feeling it now? Is there gonna be inflection? Is there any timeline predictions you wanna make? Ethan 01:29:37 : by the end of this year is-- this is going to Ethan 01:29:41 : be a big hit. So the inflection point will be where, the videos generated by video agents can get to like production grade quality, so it can be presented and it can be, it can be distributed in ads. And when-- once that happen, I think the enterprise will have much more budget for video models because the agents are, inherently more expensive than the, than the video models themselves, ‘cause they do this iterative process. They generate many variations. Ethan 01:30:23 : but once these models have this, pass this usability threshold, I think it’s, it’s going to be a exponential growth beyond that. Swyx 01:30:35 : I would, fund a company right now based on this thing. Robotics, Physical AI, and Internet-Trained World Models Swyx 01:30:40 : so I think you’re right. One thing I’m, I’m surprising, I’m reflecting on the whole like past hour or so of conversation, you are-- I think you’re into world models and video generation for video generation’s sake. I think that a lot of other world models people, we’ve interviewed a lot of them, general intuition and Fei Li and all those guys and Moondream, which I think I told you about. Moonlake. Vibhu 01:31:01 : Lake. Swyx 01:31:01 : I keep saying Moondream. Goddammit. Moonlake. A lot of them actually say like robotics is the end game. Like embodied robotics, like you want real-time, you want interactive. It is to interact with the physical world. You’re not that concerned about it. Ethan 01:31:15 : I think robotics will be a, will be a big part of it for sure.the process may happen naturally. So my prediction on robotics is that the problem is physical AI might be solved, like without actually need to Swyx 01:31:36 : Be in the real world Ethan 01:31:37 : need to be in the real world. So it might, it might get solved by a video-- A LLM is very strong video capability. So remember we talk about the real-time interactive long horizon video. Once these models-- So now these models are just training on like screen recordings and computer screens. Once these models can use computers and understand the future state of computer extremely well, the robots might be, might be one of the, one of the tools, a very powerful AI can use. So the powerful AI might just, be able to control the physical embodiment naturally. Why Ethan Left xAI and What Comes Next Swyx 01:32:28 : I see that for sure. Cool. I know, I know we are coming up on time. you had-- you left one more spicy topic, which is why you left xAI. Ethan 01:32:38 : For me, there’s, there’s a lot of, a lot of research you want to do that you cannot do at, as a company. And also like the priorities and objective the-- of a company typically can change very fast. It is-- It’s also the same for xAI. So now is kind of like the time so there is some research I want to do, especially more on language model side like I cannot do at xAI. Swyx 01:33:11 : Oh, okay, yeah. So you’re, you’re basically leaving You’re, you’re-- you had this whole transition from computer vision to world models, video generation, to now you’re like focusing on LLMs. Vibhu 01:33:22 : But it seems a lot of you saying focusing on LLMs, you really in the past hour described how it all ties together, right? Like But I don’t know. What do you mean by focusing on LLMs? Is there Ethan 01:33:33 : I realize the fact that the video models, even like in the beginning, the game might come from improvement on diffusion technology, but this is a point where actually most of the game, come from the language models themselves. Swyx 01:33:50 : It’s a huge black pill for anyone who has like spent their career in like generative, media. Vibhu 01:33:56 : it-- that’s an extreme view, right? The-- You still definitely need a bit of both, right? Vibhu 01:34:01 : There’s just, it seems like more pressing, impactful work to do now on language model side. Swyx 01:34:07 : Do you have any similar predictions? you-- so you predict the video agents, and I think you will be right. on the language side, what are you looking for in the next one year? Ethan 01:34:16 : I think one thing pretty interesting I think might be happening soon is the language models will be like context-aware and manage its own context. Ethan 01:34:29 : So some-- Like from the video model side, we’ve been suffering from the long horizon issue, like we want to generate video longer and longer, and we’ve been trying to solve the context length issues through various ways. One thing is just brute-forcing train longer context lengths. Another is to manage the context better. I think the same thing in language model is also going to be happening soon. So for example, like the language models, they’re not aware of how long their own context length is. Once they hit like eighty percent or something, automatic context compression is getting triggered. And the model, is not aware of that when it’s working. And some-- maybe it’s good for the models to know, “Oh, I’m, I’m approaching like eighty percent,” or something. And something also pretty interesting, like for example, in OpenClau, like you-- every time you type in something, a times-- the current local time is automatically attached to your message, so the model actually know what time is it. So this is making the model time-aware. And also like in tool calling the-- a lot of the intermediate tool call results automatically prune. So there’s like context removal, context addition, and, context compaction. So all of these are from the harnesses themselves. But from our experience, the heuristic engineering also helps the models get this absorbed into the models themselves. that’s something very interesting to explore. Vibhu 01:36:12 : So infinite context? Ethan 01:36:14 : Maybe. Vibhu 01:36:15 : No, but it’s, it’s interesting, right? you Swyx 01:36:17 : It is in the space of memory and continual learning and Vibhu 01:36:20 : I don’t know. It’s also like in the space of agent harness use, right? You’re seeing Swyx 01:36:25 : No, he’s saying he doesn’t want to do it in a harness, right? Vibhu 01:36:27 : No, but models are also being trained on uni-- using harnesses, right? Vibhu 01:36:32 : So some of it is, you could say, implicitly leaking in, right? part of that post-training of language models is, okay, using it in coding harnesses, in which case, when are agents spawned? When is compaction gonna happen? it’s not explicit you have this much token window, which I don’t know if you want it to be, as that’ll change, but it’s, it’s somewhat leaking in there. Ethan 01:36:58 : I’m imagining, what if the model have access to the whole-- the code of the agent harness itself and being able to modify it to whatever you want. Say, if the agent harness is short enough, you can just put in the context lengths in the system prompt, and then the model will say, “When I want to spawn a future version of myself, I can modify the agent harness.” For example, if I-- the agent harness can be, “Oh, when I’m reading-”A long document, I can choose to read the whole thing in chunks and, come back, smash the summary together, or I can just read the first two hundred lines and, discard the rest. And all kind of choices, if they can be made by the models themselves, it might be very interesting to see that the model can, program the model can program itself online in test time. Career Lessons: Moving Across ML Domains Swyx 01:38:02 : so the self-modifying harness is also part of, OpenClaw and Py, but I think there’s a lot more work to do there. Very cool. I think part of me is kind of curious. I think you are part of Big Lab, right? And there’s this career path of a researcher at a Big Lab, which is you are-- you train models, you get more compute, you train better models, and you keep going. And somewhat, I feel like you’re opting out of that. And if I were you, I would be “Oh, I think this is, a bit of a career risk.” what? Swyx 01:38:36 : I don’t have any comment apart from, you’re very strongly convicted. I think that a lot of people in your shoes would not be doing what you did. Ethan 01:38:43 : Speaking of my career, if I look back, actually, there were, there were a lot of huge transitions. So ten years ago, I was, I was doing research with a ResNet authors, Xiangyu Zhang and Jian Sun. Yeah, at that time, the research were completely different. It was, mostly confirmation, like image recognition, object detection, object tracking. I was also doing neural net compression at that time. It was quite different from knowledge dissolutions these days. And at that time, I was-- I wanted to be a professor, and I applied. When I applied for a PhD, I already had a few first author papers at top conferences, so I confidently applied at the top schools. It turns out I got rejected by all of the top PhD programs. So I had to, I had to go to the industry. At that time, I was at Facebook AI Research fair, led by Yann LeCun. Swyx 01:39:51 : I wanted to talk about VJPA, but it’s different. Ethan 01:39:53 : I know. Yeah, we can leave it for another time. Ethan 01:39:57 : I switched to At that time, I switched to self-surprised learning. It was, it was quite different from what I was doing in contribution. Ethan 01:40:07 : And after that is NVIDIA Cosmos. So I realized scaling up was extremely important. So at NVIDIA, I was mainly focusing on scaling. So one thing is Cosmos scaling the video distribution models to a few billion parameters. And another thing is, I was working on MoEs. The Megatron MoEs was the first, was the first framework open source to be able to train these MoEs at very large scales, hundred billions parameters to even trillions parameters efficiently at, forty percent MFU. Ethan 01:40:51 : And going to switching to xAI was trying to work on even larger compute scale even further. And yeah, looking at this trajectory, I actually worked on a lot of different things. So I feel actually within ML, it’s actually easier to switch than you think. a lot of people might have mindset that, “Oh, I work on, I work on computer vision. I always have to work on computer vision, and I cannot switch to language.” And, but from my experience, at least at NVIDIA, I worked on both language model MoEs and also video models. It’s, it’s actually not the case. A lot of, a lot of the core principles how to train large models are largely the same. And yeah, for me, I feel right now the bottleneck, for video models is actually the language part the agent, which is why I want to go to work more on LLMs. One thing is it’s, it’s a bit of a challenge. I don’t think it’s a huge, jump, so. Closing Thoughts Swyx 01:42:18 : kudos to you. I think you have a lot of, strong vision there. Yeah, I think that was mostly everything that we wanted to cover. You’ve been very generous with your time, and I, it’s really nice that you are able to share all these things now. We don’t have to go through xAI to clear everything. but also we Ethan 01:42:35 : Oh, Swyx 01:42:35 : I think we didn’t get you in trouble. Ethan 01:42:37 : It’s a lot of good stuff about xAI compared to what you just see in the releases, right? You don’t realize how many more levels there are to it. Swyx 01:42:44 : xAI, please do more podcasts. Swyx 01:42:47 : anyway. Swyx 01:42:48 : but thank you for, sharing. It’s been very kind. And also, I wanna hear more from you. I think you are going to embark on your next phase. You haven’t announced what you’re doing next, but clearly you have, more vision and more ambition on this path, and I think you’re, you’re basically kind of gradient descending to, whatever your final form is. Ethan 01:43:08 : Thank you. Yeah. Yeah, I’ll, I’ll share more about my next chapter soon. Ethan 01:43:14 : Thank you for having me. Swyx 01:43:16 : Thanks for coming.