According to the Hugging Face model card, the NTU-led model generates camera-controlled RGB and depth views that can be consolidated into 3D point clouds.
By RuntimeWire Staff · Published
Primary source: Hugging Face Newsroom
Why it matters #
Puffin-World makes camera pose and depth first-class model states, a useful direction for robots and agents that need stable 3D environments instead of convincing video alone.
Kang Liao and nine other researchers detailed the Puffin-World project on September 2, giving multimodal models explicit representations of gravity, camera orientation and scene depth instead of asking generated pixels to carry all the spatial information. The group also released model checkpoints, a public code repository, its Puffin-16M training dataset and supplementary evaluations.
Liao, a research fellow at NTU's MMLab and S-Lab, has spent much of his research career on the camera problems that image generators tend to conceal. His earlier work covered image stitching, panoramic depth estimation and image restoration before he moved into unified multimodal models. Puffin-World extends the earlier Puffin project, which was accepted to ICLR 2026, from camera-aware image understanding and generation into native 3D world states.
The other authors are Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li and Chen Change Loy. Loy directs MMLab@NTU, while Wu is an NTU Ph.D. student working on embodied AI and vision-language-action models.
Teaching a model which way is up
Puffin-World starts from a practical weakness in generative video and world models: a sequence can look plausible frame by frame while losing track of the camera's relationship to the physical scene. Large rotations and long trajectories expose the problem. Horizons drift, upright objects tilt and separate views stop agreeing on a stable world frame.
Liao's team represents each scene through three native states. The physics state contains a gravity field and latitude map, which indicate scene uprightness and the elevation of viewing rays. The geometry state uses depth. The appearance state contains RGB images and sequences. According to the Puffin-World model card, the model predicts and generates these states together, then uses the resulting multi-view RGB and depth output for 3D point-cloud reconstruction.
The central mechanism, called Omni-Camera, gives every pixel a nine-channel camera condition. Three channels describe an absolute field containing an up vector and latitude angle. Six more describe relative ray origin and direction. That pairing lets the model preserve a global reference to gravity while moving continuously between viewpoints.
Starting from one image or a text prompt, the model can generate new views along camera trajectories that include translation, roll, pitch, yaw and compound motion. It can also estimate camera parameters from an uncalibrated image, generate depth alongside appearance and consolidate the output into a 3D reconstruction without a separate offline geometry system.
That combination is the team's core bet. World models intended for embodied agents need an internal account of camera pose and spatial structure. Photorealistic frames alone provide a weak foundation when an agent must revisit a location, rotate sharply or reason about what lies outside its initial view.
Sixteen million camera examples
The training effort centers on Puffin-16M. Its Puffin-Cam-15M portion contains 15 million vision-language-camera triplets rendered from roughly 900,000 panoramas. A second portion, Puffin-Traj-1M, contains 1 million trajectories covering upward and downward motion, clockwise and counterclockwise rotation, full 360-degree exploration and compound camera movements. Puffin-Traj-1M supports camera-controlled world exploration and trajectory-aware generation and understanding.
The researchers also used Puffin-World to annotate about 44.5 million images from 28 public datasets with roll, pitch and vertical field-of-view information. According to the team's September release, those annotations are intended to help researchers analyze viewpoint bias and curate camera-diverse training sets.
The model card reports the best median errors in 12 of 12 camera-understanding comparisons, including ties, across Stanford2D3D, MegaDepth, TartanAir and LaMAR. It also reports first-place PSNR and LPIPS results on RealEstate10K. Those numbers come from the authors' own evaluation materials and have not been independently replicated in the released reporting.
The available checkpoints reflect different tradeoffs. The Puffin-World model card says Base combines a Qwen2.5-7B language model, C-RADIOv3-H vision encoder and Stable Diffusion 3.5 Medium. Pro uses a smaller Qwen2.5-1.5B language component with C-RADIOv4-H and Stable Diffusion 3.5 Large. A captioning checkpoint uses Qwen3.5-0.8B and C-RADIOv3-H without a diffusion component.
A research release, not a service
Puffin-World is an academic research release distributed through Hugging Face, GitHub and the project website rather than a commercial service.
For Liao and his collaborators, the September release turns years of camera-centric research into a downloadable stack that other labs can inspect, run and challenge. The group has put the code, weights and data in public view. The next evidence will come from researchers trying to reproduce the reported gains and testing whether Puffin-World's stable geometry survives beyond its curated demonstrations.