Real-time prompt-based stream editing for AR at 1080p, 46 FPS and 87 ms latency

A researcher received the Best Demo Award (Honorable Mention) at IEEE ISMAR 2026 in Bari, Italy, for "Flow Matching for Augmented Reality Applications," a real-time prompt-based AR stream editing demo that renders at 1080p, 46 FPS and 87 ms latency. The demo argues diffusion models are a feasible alternative to Gaussian Splatting and Neural Radiance Fields for AR, and was built on Chinese open-weight models from Alibaba and Z.ai, roughly 1000 GB200 hours on France's DALIA cluster of 64 NVIDIA GB200 GPUs, and Andrej Karpathy's autoresearch for code-space parameter search. The researcher reported limitations including temporal inconsistencies and more failed edits as the number of people in frame increases.

Demo Award at IEEE ISMAR 2026 At the International Symposium on Mixed and Augmented Reality IEEE ISMAR in Bari, Italy, I received Best Demo Award Honorable Mention for: Flow Matching for Augmented Reality Applications . In this post, I first highlight the demo’s capabilities and limitations. Second, I explain how I made it; I trust that this will be useful for other practitioners 1. The Effect A lot of researchers in the AR and VR communities are investigate rendering with Gaussian Splatting https://en.wikipedia.org/wiki/Gaussian splatting and Neural Radiance Fields https://en.wikipedia.org/wiki/Neural radiance field . I believe that these techniques are fine for VR and pure computer graphics, but are completely wrong for AR. This was one of the main motivations for the demo: show that Diffusion Models are a feasible alternative that deserve careful consideration. The quickest way to familiarize yourself with the high-level story is to watch this 1-minute explanatory video: At the conference, I recorded the screen of my laptop while several attendants were playing with the demo in order of appearance: Ke Li https://www.inf.uni-hamburg.de/en/inst/ab/hci/people/keli.html , Doron Friedman https://www.runi.ac.il/en/faculty/doronf , Xinyi Ye https://www.hitlabnz.org/index.php/people/ student , Huyen Nguyen https://ar-ai.org/author/huyen-nguyen/ , and Yingfangzhong Sun https://ar-ai.org/author/yingfangzhong-sun/ : Unfortunately, the screen recording only ran at 7 FPS at 720p 🤦🤦. In order to get closer to the actual rendering performance, I performed post-processing with the FILM https://film-net.github.io frame interpolation model to get to 30 FPS, which led to melting artifacts that were not present in the actual demo. Effects that can be observed include: - Virtual light sources illuminate real elements - Virtual elements are registered in the real world - Several prompts are stacked successively There are also defects visible that are inherent to my pipeline: - Temporal inconsistencies - The more people are in the frame, the more edits fail I had great fun playing together with one of my PhD advisors: Steven Feiner https://www.cs.columbia.edu/~feiner/ . Moronically, I only captured the output, but not the input 🤦. I used Qwen-Image-2.1 https://huggingface.co/Qwen/Qwen-Image-2.1 to recreate the “before” frame. 2. How it’s Made Three Key Ingredients This demo would have been hardly possible without Chinese open-weight models , generously donated to the public by Alibaba Inc. and Z.ai. 谢谢 For reasons that become apparent further down, closed models, such as Anthropic or OpenAI, would not have been feasible to use within a reasonable financial or time budget—they are simply too slow and too expensive for the approach I took One of the biggest perks of being a researcher in France is having access to our DALIA GPU beast http://www.idris.fr/en/docs/dalia/dalia-presentation/ , which has 64 NVIDIA GB200 GPUs. For lighter tasks, I used the older Jean-Zay http://www.idris.fr/en/docs/jean-zay/jean-zay/jean-zay-hardware gpu-nvidia-h100-partition-gpu p6 with 1456 H100 GPUs. I estimate that I have used the equivalent of 1000 GB200 hours for developing this demo. The final key ingredient is Andrej Karpathy’s autoresearch https://github.com/karpathy/autoresearch , which transposes the classical idea of automatic parameter search into code space . I used this mechanism on several levels of abstraction when developing the demo. Development Stack The summary of my development stack from bottom to top is: - vllm https://vllm.ai on DALIA for high-performance serving of LLMs. I used autoresearch to optimize startup times e.g. GLM went from 1 hour to 8 minutes , and inference speed. - Pi https://pi.dev as agent harness - Herdr https://herdr.dev to manage agent swarms - Moshi https://getmoshi.app as mobile phone client so that I could manage my agents from everywhere Let’s spend a moment to look at the most fundamental part of this stack: the LLMs used and their performance. In terms of speed, I can use leading open weight models at about 10x speed compared to OpenAI models and 5x faster than Anthropic models: The main autoresearch loop for the demo ran about 1 month continuously. A 10x slower model such as Anthropic’s Fable would have simply not been feasible. But hold on Aren’t the closed source models smarter? Indeed, they are: A problem that requires an intelligence index over 50 would lead to open weight models running in high-speed circles without ever solving them To my great delight, I have not yet encountered such a problem. Demo Development To attack the challenging performance issues, I ran autoresearch loops for individual components of the inference pipeline DIT, VAE, Upscalers , as well as the overall system. Key instructions included: - Increase FPS - Decrease Latency - Verify that visual quality is preserved using a test suite with a VLM Some of the most impactful optimizations happened on the architectural level, such as cross-GPU pipeline parallelism: The agents discovered an astonishing amount of bugs in NVIDIA’s TensorRT for Blackwell. For example, see above diagram, it was not possible to use TensorRT TRT for VAE encoding/decoding at my demo’s resolution it only worked for smaller ones ; so we had to fall back to the much slower torch implementation. Luckily the DIT, RIFE, and ESRGAN components worked with it ;- Show Time 3. Conclusions I am still trying to fully wrap my head around this development experience It is certainly very different from anything I have ever done in my 31 years of programming A mini-postmortem: - Autoresearch is unreasonably effective. - 1000 GPU hours were wasteful. I should have planned better. - Agent swarms on multiple heterogeneous machines are not trivial. Especially for network optimizations, it was useful to have agents on DALIA communicate with agents on my laptop, which regularly ran into the aggressive intrusion detection system on DALIA. In the end, I figured everything out, even how to make persistent connections between moshi on my phone and DALIA. But, this took me too much time. What comes next? This is a big question that I need to think deeply about. The only thing that is already clear to me is that the next experiment will use about 10x more compute :- You can find some more information about the demo in the Demo description as submitted to ISMAR 2-page pdf paper.pdf . Please note that the performance numbers are outdated I worked a few more weeks on the demo after submission .