GaussianGPT: Generating 3D scenes autoregressively with Gaussian Splatting

Researchers from the Technical University of Munich introduced GaussianGPT, a transformer-based model that generates 3D scenes autoregressively via next-token prediction on Gaussian splatting tokens, accepted as an oral presentation at ECCV 2026. The model compresses 3D Gaussians into discrete tokens using a sparse 3D convolutional VQ-VAE and generates scenes step-by-step, supporting completion, outpainting, and controllable sampling. The full training and inference code, along with pre-trained checkpoints, were released on GitHub.

Nicolas von Lützow · Barbara Rössle · Katharina Schmid · Matthias Nießner Project Page https://nicolasvonluetzow.github.io/GaussianGPT/ · arXiv https://arxiv.org/abs/2603.26661 · Paper https://arxiv.org/pdf/2603.26661 · Video https://youtu.be/zVnMHkFzHDg Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. We instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation. We first compress Gaussian primitives into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding, enabling sequential generation of spatial structure and appearance. Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons. This formulation leverages the compositional inductive biases and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation. 2026-08-05 - GaussianGPT selected for an oral presentation 2026-07-12 - Object-level PhotoShape checkpoints checkpoints released. 2026-07-02 - Camera-ready paper now available on arXiv https://arxiv.org/abs/2603.26661 . 2026-07-01 - Pre-trained scene-level VQ-VAE and GPT checkpoints checkpoints released. 2026-06-19 - Training and inference code released. 2026-06-18 - GaussianGPT accepted to ECCV 2026 This repository contains the full training and inference code for GaussianGPT. The pipeline is two trained models connected by a tokenization step: VQ-VAE train ae.py — a sparse 3D CNN with vector quantization that compresses per-voxel Gaussians into discrete tokens, supervised by a gsplat re-rendering loss. Tokenization tokenize dataset.py — runs the trained encoder over the dataset and writes per-scene token streams to disk. GPT train gpt.py — an autoregressive transformer with 3D rotary embeddings that models the token streams via next-token prediction. Inference — sample from the GPT and decode through the frozen VQ-VAE to generate, complete, or tile Gaussian scenes, then render them. php Gaussians --VQ-VAE-- tokens --GPT-- sampled tokens --decode-- Gaussians --render-- Everything is configured with Hydra https://hydra.cc/ ; override any field on the CLI as key=value . If you find GaussianGPT useful, please consider citing: @inproceedings{vonluetzow2026gaussiangpt, title = {GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation}, author = {von L{\"u}tzow, Nicolas and R{\"o}{\ss}le, Barbara and Schmid, Katharina and Nie{\ss}ner, Matthias}, booktitle = {European Conference on Computer Vision ECCV }, year = {2026}, } The environment is built around CUDA 12.9 and PyTorch 2.8. Several dependencies are compiled from source, so make sure a GPU is visible during installation for correct CUDA support. Set the target architectures before any from-source build. Refer to the NVIDIA GPU feature list https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html gpu-feature-list for your hardware. export TORCH CUDA ARCH LIST="8.6;8.0" e.g. Ampere 3090, A6000, A100 conda create -n gaussiangpt python=3.10 nvidia::cuda-toolkit=12.9 conda-forge::glm ninja conda activate gaussiangpt Point CUDA HOME at the conda toolkit persist for future sessions and apply now . ln -s "$CONDA PREFIX/lib" "$CONDA PREFIX/lib64" conda env config vars set CUDA HOME="$CONDA PREFIX" export CUDA HOME="$CONDA PREFIX" pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu129 pip install --no-build-isolation -r requirements.txt compiles several extensions; slow requirements.txt installs two CUDA extensions from pinned git commits that are compiled from source during this step hence the GPU-visible requirement and the long build time : gsplat https://github.com/nerfstudio-project/gsplat the re-rendering loss / renderer and rotation/quaternion ops in https://github.com/facebookresearch/pytorch3d pytorch3d utils/transforms.py . If the install fails, it is almost always one of these two — build them individually to see the full nvcc error, and make sure TORCH CUDA ARCH LIST and CUDA HOME are set as above. git clone https://github.com/Dao-AILab/flash-attention.git cd flash-attention git checkout v2.8.2 MAX JOBS=4 python setup.py install Troubleshooting: MAX JOBS=1 Each nvcc job needs several GB of RAM. Start MAX JOBS low and raise it only if you have the RAM to spare; lower it down to 1 if the build OOMs / gets "Killed". Optional: FLASH ATTN CUDA ARCHS=80 flash-attn compiles its own arch set sm 80;90;100;120 and ignores TORCH CUDA ARCH LIST . Its Ampere kernels are sm 80, which also run on sm 86 3090/A6000 via same-major forward-compat, so restricting to sm 80 covers all the target GPUs above and cuts both compile time and per-job RAM substantially. The upstream MinkowskiEngine does not build against CUDA 12 without source patches. The alpsaur fork https://github.com/alpsaur/MinkowskiEngine/tree/cuda12-compat bundles the CUDA 12 fixes, so no manual patching is needed: git clone https://github.com/alpsaur/MinkowskiEngine.git --branch cuda12-compat cd MinkowskiEngine conda install -c conda-forge "blas= =openblas" openblas openblas-devel python setup.py install --blas include dirs="${CONDA PREFIX}/include" --blas=openblas Compiler note: build with GCC <= 13 MinkowskiEngine does not compile with GCC = 14 against PyTorch 2.8's bundled pybind11 — you'll hit ambiguous template instantiation errors on the enum registrations. Use GCC <= 13 GCC 11 is known good . The openblas-devel install above can pull a newer GCC from conda-forge into the env, so after running it check ${CXX} --version ; if it reports 14+, pin an older toolchain and reactivate so conda repoints CC / CXX before building: conda install -c conda-forge gcc linux-64=11 gxx linux-64=11 conda deactivate && conda activate gaussiangpt reactivate to refresh CC/CXX If conda balks at re-solving cuda-toolkit , pin it explicitly and freeze the rest: conda install -c nvidia -c conda-forge cuda-toolkit=12.9.2 gcc linux-64=11 gxx linux-64=11 --freeze-installed . Alternative: official repo with manual patches Clone upstream and apply the CUDA 12 source patches yourself, following issue 543 https://github.com/NVIDIA/MinkowskiEngine/issues/543 issuecomment-1773458776 : git clone https://github.com/NVIDIA/MinkowskiEngine.git cd MinkowskiEngine Apply the CUDA 12 source patches from the issue above before building. conda install -c conda-forge "blas= =openblas" openblas openblas-devel python setup.py install --blas include dirs="${CONDA PREFIX}/include" --blas=openblas Optional: per-voxel dedup torch scatter Not required for the default pipeline. Only needed if you enable model.make unique=true collapse each voxel to its highest-opacity Gaussian , which is off in the shipped configs. The import is lazy, so the dependency is only touched when that option is turned on. In our tests it gave a negligible quality gain for the added compute. pip install --no-build-isolation torch-scatter -f "https://data.pyg.org/whl/torch-2.8.0+cu129.html" We are grateful to the authors of the following datasets, whose data made this work possible. Please refer to the respective sources for licensing and download instructions. | Dataset | Source data | Gaussians | |---|---|---| | PhotoShape | | original data https://www.projectaria.com/datasets/ase/ SceneSplat-49k https://huggingface.co/datasets/GaussianWorld/aria synthetic envs mcmc 3dgs new original data https://tianchi.aliyun.com/specials/promotion/alibaba-3d-scene-dataset no longer available 3D-FRONT availability The original source data is no longer available, but more recent re-releases e.g. this one https://huggingface.co/datasets/huanngzh/3D-Front should work similarly. Data format PhotoShape — standard Inria-style 3DGS .ply files y-up . 3D-FRONT — PyTorch dicts; easiest to follow the data-loading code directly. The dicts hold the default 3DGS attributes, but split position into anchor 3D anchor positions and offset . f dc / f rest hold the SH coefficients; all other attributes are stored pre-activation logits . Voxel-GS preprocessing Voxel-GS is the simplified Scaffold-GS https://city-super.github.io/scaffold-gs/ from L3DG https://barbararoessle.github.io/l3dg/ — no MLP, one Gaussian per voxel, no hierarchy. PhotoShape — L3DG Voxel-GS as-is paper Secs. 3.2.1, 3.4 . 3D-FRONT — same, with: point cloud from back-projected depth maps; no scene normalization; 2.5 cm voxels; anchor densification off; 60k iterations; SH degree capped at 1; scales bounded by 2 voxel size sigmoid . Training is a two-stage pipeline VQ-VAE, then GPT with a tokenization step in between. Checkpoints and logs land under experiment.log dir logs/ by default , organized by experiment.name . The dataset configs reference scene-name lists under data splits/ . Two standalone helpers under scripts/ build them in two steps — scan once to produce a per-scene stats CSV, then filter that CSV into train/val splits re-run the cheap second step with different thresholds without re-scanning : python scripts/dataset quick stats.py \ --data-root <gaussians root --transforms-root <transforms root \ --output logs/stats.csv python scripts/dataset split from quick stats.py \ --input logs/stats.csv \ --train-split data splits/train.txt --val-split data splits/val.txt \ --min-images 100 --max-points 5000000 \ --min-extent-x 3.2 --max-extent-x 25 \ --min-extent-y 3.2 --max-extent-y 25 --max-extent-z 4 All filters default to off; pass --help on either script for the full set. Trains the sparse-CNN autoencoder with vector quantization that compresses per-voxel Gaussians into discrete tokens. Reconstruction is supervised by a re-rendering loss gsplat . python train ae.py \ data=vfront houses \ experiment.name=my vqvae - Top-level config: conf/vqvae.yaml data=vfront houses , model=vqvae cnn , training=vqvae . Swap the dataset with data=photoshape , data=ase , data=spp v2 , etc. - Loss weights live in conf/training/vqvae.yaml ; the defaults target VFront/ASE, and the file notes the PhotoShape overrides. - PhotoShape additionally needs model=vqvae photoshape finer voxels, view-dependent color . The training overrides are listed at the top of that config. max epochs is a deliberate overestimate — stop the run by picking a checkpoint rather than waiting for it to finish.- Resume with experiment.checkpoint path=<ckpt experiment.continue mode=resume or weights only to load only the weights and reset the optimizer/scheduler . Runs the trained VQ-VAE encoder over the dataset and writes per-scene token streams to disk. The output directory becomes data.data path for the GPT stage, and the VQ-VAE checkpoint becomes data.vqvae path . python tokenize dataset.py \ data=vfront houses \ experiment.checkpoint path=<vqvae.ckpt \ training.tokenization.output dir=<tokens dir - Shares the conf/vqvae.yaml config; only experiment.checkpoint path and training.tokenization.output dir are required both are asserted at startup . training.tokenization.sort by controls latent ordering; training.tokenization.generate augmented samples=true writes 8 variants per scene 4 z-rotations x mirrored/not . Trains the autoregressive transformer prior over the VQ tokens. The frozen VQ-VAE is loaded from data.vqvae path and used only for decoding during the inline evaluation. python train gpt.py \ data=tokenized vfront \ data.data path=<tokens dir \ data.vqvae path=<vqvae.ckpt \ experiment.name=my gpt - Top-level config: conf/gpt.yaml data=tokenized vfront , model=gpt , training=gpt . Use data=tokenized vfront ase for the combined VFront+ASE model. data path and vqvae path are required ??? in the tokenized data configs and must be supplied. - For the object-level model use data=tokenized photoshape model=gpt photoshape GPT-2-small on a 32^3 grid . - Transformer size is set inline via the gpt size block in conf/model/gpt.yaml n embd , n layer , n head , n kv head . The default n embd=1024 , n layer=24 matches GPT-2-medium. - Multi-GPU is auto-detected: the run uses ddp when more than one GPU is visible. - Evaluation runs inline via EvaluateCallback , cadenced by training.output.render frequency . training.output.eval data config selects which raw conf/data/ .yaml is composed for the rendering. experiment.continue run=true auto-finds the latest checkpoint from a prior run with the same log dir / name . Pre-trained VQ-VAE and GPT checkpoints are hosted at kaldir.vc.cit.tum.de/gaussiangpt sizes and SHA256 checksums in the served README https://kaldir.vc.cit.tum.de/gaussiangpt/README.md . Each GPT must be paired with the VQ-VAE listed alongside it. Trained on 3D-FRONT wget https://kaldir.vc.cit.tum.de/gaussiangpt/vqvae vfront.ckpt wget https://kaldir.vc.cit.tum.de/gaussiangpt/gpt vfront.ckpt Pre-trained on 3D-FRONT + ASE, fine-tuned on 3D-FRONT wget https://kaldir.vc.cit.tum.de/gaussiangpt/vqvae both.ckpt wget https://kaldir.vc.cit.tum.de/gaussiangpt/gpt both.ckpt Trained on PhotoShape object-level wget https://kaldir.vc.cit.tum.de/gaussiangpt/vqvae photoshape.ckpt wget https://kaldir.vc.cit.tum.de/gaussiangpt/gpt photoshape.ckpt Pass them to any inference entry point as checkpoint=<gpt.ckpt vqvae checkpoint=<vqvae.ckpt see Inference inference . All inference entry points take the trained GPT via checkpoint=<gpt.ckpt and need a VQ-VAE checkpoint to decode sampled tokens. The VQ-VAE is resolved as vqvae checkpoint=<vqvae.ckpt explicit override → model.vqvae.checkpoint path → data.vqvae path from a composed data=<cfg . The token streams data.data path are only read when conditioning on real scenes completion ; unconditional sampling does not touch them. Samples one chunk per scene, decodes through the VQ-VAE, and renders camera trajectories. This module also provides the inline helpers imported by train gpt.py . For object-level models PhotoShape a chunk is a whole object, so this is the full object-generation pipeline; the completion and multi-chunk entry points below are scene-level. Unconditional generation: only the GPT and a VQ-VAE checkpoint are needed. python generate chunks.py \ checkpoint=<gpt.ckpt \ vqvae checkpoint=<vqvae.ckpt \ num samples=4 temperature=1.0 Key options see conf/generate chunks.yaml , the config generate chunks.py loads : num samples , batch size , temperature / top k / top p , seed , store samples , render gifs set render gifs=false to skip GIF rendering, e.g. to only dump samples via store samples=true . Outputs go to output dir outputs/eval by default . Prompt-conditioned completion sequence-prefix sanity check generate chunks.py can also condition on a real scene by enabling completion : it keeps the first prompt fraction of the token sequence and continues it, producing one completion per scene. This is a quick sanity check on the prior, used primarily during training. For completion cut by spatial extent , use complete chunks.py below. python generate chunks.py \ checkpoint=<gpt.ckpt \ data=tokenized vfront \ data.data path=<tokens dir data.vqvae path=<vqvae.ckpt \ completion.enabled=true completion.prompt fraction=0.5 The prompts are read from the tokenized data=<cfg completion.split selects train/val ; prompt fraction is a continuous float default 0.5 . Conditions on part of a scene and samples the rest — the autoregressive analogue of inpainting/outpainting. The prompt is cut by spatial extent e.g. keep half the room along x rather than by sequence length, and several completions are sampled per scene. python complete chunks.py \ checkpoint=<gpt.ckpt \ data=tokenized vfront \ data.data path=<tokens dir data.vqvae path=<vqvae.ckpt \ prompt mode=spatial half x Key options see conf/complete chunks.yaml : split , prompt mode , num completions , num samples , the sampling params, and render gifs / store tokens . Supports sharding via shard id / num shards . Outputs go to outputs/complete chunks . generate scene.py autoregressively tiles many chunks into a large scene, then decode scene.py turns the saved token sidecars into renderable Gaussian payloads. generate scene.py shards over the tile grid via the GAUSS SHARD ID / GAUSS NUM SHARDS env vars, so it runs cleanly as a SLURM array — each task only reads/writes its own rank XXXX shard. Sample single shard shown; for a SLURM array set the two env vars per task . GAUSS SHARD ID=0 GAUSS NUM SHARDS=1 python generate scene.py \ checkpoint=<gpt.ckpt \ vqvae checkpoint=<vqvae.ckpt \ num scenes=4 output dir=<scene out Decode the token sidecars into Gaussian scenes. python decode scene.py \ --output-dir <scene out \ --vqvae-checkpoint <vqvae.ckpt generate scene.py can decode and render top-down inline decode outputs , render topdown , both on by default ; see conf/generate scene.yaml for the tiling scene cols x/y , the bootstrap/outpainting sampling params, and the empty-column handling. decode scene.py infers the GPT checkpoint from the run manifest, so only --output-dir and --vqvae-checkpoint are required. Standalone renderers operate on decoded scene .pt payloads keys coords , sh0 , opacities , scales , quats , plus optional sh with higher-order SH coefficients, rendered when present : Single top-down PNG. python render topdown.py --input <scene.pt --quantile 75 --resolution 1024 Rotating-orbit GIFs for one scene or a directory of scenes. python render orbit batch.py --input <scene or dir --output-dir <render out Object samples PhotoShape are y-up; keep the full object and orbit from outside. python render topdown.py --input <sample.pt --up y --quantile 100 python render orbit batch.py --input <samples dir --output-dir <render out --up y --move-back 1.75 Payloads keep their dataset's native up axis: 3D-FRONT/ASE are z-up the default , PhotoShape is y-up pass --up y to rotate before rendering . render topdown.py drops points above --quantile to see through ceilings; use --quantile 100 to keep whole objects . render orbit batch.py accepts a single .pt or a directory --max-files caps how many it processes and writes per-frame images plus a GIF under --output-dir . Its default --move-back orbits from inside a room; use ~1.5-3 to orbit around an object. To inspect a payload in an external viewer, scripts/convert pt to ply.py converts a Gaussian .pt / .pth payload to an INRIA-style .ply . This work would not have been possible without the following open-source projects, and we thank their authors and contributors. This project is released under the MIT License. See LICENSE /nicolasvonluetzow/GaussianGPT/blob/main/LICENSE for details.