# Gaussian Splatting for Self-Driving Simulation, Built in Plain PyTorch on a Laptop GPU > Source: > Published: 2026-10-19 20:01:01+00:00 Author: Tamoghna Das Autonomous driving software is extensively tested in simulation because it enables testing in various dangerous situations. The simplest simulator replays a recorded log, allowing the autonomous driving planner to see what the car’s camera saw. But as soon as the simulated car driven by the planner does something different than the recorded car, it has to see the road and the environment from a point of view that’s not in the log. The 3D Gaussian Splatting (3DGS) [1] technique allows scene rendering from new viewpoints. Driving-scene methods build on it: Street Gaussians [4] separates static streets from moving cars, and SplatAD [5] renders LiDAR together with cameras. Almost all 3DGS implementations use custom CUDA kernels for the heavy lifting. That makes them fast, but it also makes the core of the method opaque, impossible to step through with a Python debugger, and tied to the NVIDIA driver and a compiler toolchain. *gsplat* (v1.5.3) comes with a PyTorch reference rasterizer for debugging, but it still calls a CUDA kernel to calculate which Gaussians touch which pixels, needs the *nerfacc* package and, according to its documentation, is much slower and uses much more GPU memory. This experiment takes the kernels away. The entire pipeline of 3DGS with training and rendering is implemented in pure PyTorch on a real driving environment on a modest NVIDIA RTX A1000 laptop GPU with 4 GB of VRAM. The training environment has no *gsplat* and no CUDA toolkit. PandaSet [2], a driving dataset, has 8-second drive scenes recorded with six cameras, a 360° spinning LiDAR and a forward-facing long-range LiDAR. Objects are annotated with 3D boxes. The model in this article is trained on images from three forward-facing cameras (front, front-left, front-right) and on data from both LiDARs. I chose PandaSet sequence 28 because it has the fewest moving objects around the car. It is an 84-meter drive up a residential street with parked cars and trees along the roadside. Every 8th frame is held out for testing. The LiDAR data provides surface points, which give the Gaussians their starting positions and colors. Also, LiDAR data projected onto each training photo provides the true distance at many of its pixels, which is used to check the model’s rendered depth during training. Every photo and render has license plates and people blurred. The model represents the entire 84-meter street (road, parked cars, trees, building fronts) with a million 3D Gaussians: soft, semi-transparent, colored ellipsoids hanging in space. Each Gaussian is defined by its position, size along three axes, rotation, opacity and color that can vary slightly with viewing direction. Training involves feeding the model one training camera frame at a time. From that frame’s viewpoint, every Gaussian is projected onto the image plane as a 2D ellipse, sorted by distance and blended front to back into an image (this is rendering or rasterizing). The rendered image is compared with the real training image. Gradient descent updates every Gaussian’s parameters to minimize the error. Training also adds Gaussians where details are missing and drops Gaussians that become transparent. Once the training is over, the trained Gaussians represent the scene. Rendering a frame at inference time needs only a camera position, orientation and lens. Gaussians are projected, sorted and blended to form a novel view, such as a perspective from a lane not in the training data. The renderer follows a four-step process (project, bin, sort, composite) to form an image from a camera point of view. This implementation keeps the same four stages as the CUDA rasterizer but parallelizes them the way tensors like. **Project**: Each 3D ellipsoid becomes a 2D ellipse, smaller the further away the 3D ellipsoid is from the image plane. Gaussians behind the camera or entirely off-screen are discarded. **Bin**: The image itself is cut into 8×8-pixel tiles. I calculate which tiles each 2D projected ellipse touches. Each (Gaussian, tile) pair becomes one entry. **Sort** (without a CUDA kernel): The entries are grouped by tile, and within each tile ordered by depth: each Gaussian’s distance from the camera. This is important because each Gaussian’s contribution to a pixel is proportional to how much light the Gaussians in front of it let through. From *splat/render.py*, inside *bin_and_sort()*: ``` # Depth order first; a stable sort by tile afterwards keeps it inside every tile.order = torch.argsort(depths)counts_o = counts[order]total = int(counts_o.sum().item())gauss = torch.repeat_interleave(order, counts_o) # (M,) one entry per pairfirst = torch.cumsum(counts_o, 0) - counts_o # where each Gaussian's run startsk = torch.arange(total, device=means2d.device) - torch.repeat_interleave(first, counts_o)w = nx[gauss]tile_ids = (ty0[gauss] + k // w) * tiles_x + (tx0[gauss] + k % w)tile_ids, perm = torch.sort(tile_ids, stable=True)gauss = gauss[perm] ``` In the code snippet above, *argsort(depths)* implements the sort by depth: it puts every Gaussian in order, nearest first. *repeat_interleave* implements the binning: it copies each Gaussian once for every tile it touches. *sort(tile_ids, stable=True)* groups the entries by tile. “*Stable*” means entries with the same tile keep their earlier order, so each tile’s list is already front to back. **Composite**: After sorting, every tile has its own front-to-back list of Gaussians. Stacking them all into one tensor would mean around 2.6 GB for a single tensor. On the other hand, processing one Gaussian at a time, as the CUDA kernel does per pixel, would mean thousands of small steps launched from Python. The CUDA kernel runs one thread per pixel. I implement the rendering calculation in chunks. Each round, the renderer takes the next few Gaussians (one chunk) of every unfinished tile, front to back, computes their effect on all 64 pixels (8×8 pixels per tile), and saves how much light still gets through each pixel for the next round. Since *gsplat* [3] is the de facto reference, a frame produced by the rasterizer is compared pixel-by-pixel with a *gsplat*-produced frame. On random scenes of up to 100,000 Gaussians, the mean absolute pixel difference from *gsplat* is at most 6.7 × 10⁻⁵ on a 0–1 color scale. On the trained street scene, the two rendered images agree to a median PSNR of 66.3 dB. Training follows the 3DGS paper, with a few adaptations for driving data recorded by three forward-facing cameras and two LiDARs. Training starts with 400,000 Gaussians initialized from LiDAR points. To check the training result, I render the 30 held-out frames (every 8th frame of each of the three cameras), which were not used in training, and compare them with real photos using three standard image scores: After 30,000 training steps (119 minutes), the model representing the 84 m street has just under one million Gaussians and scores a PSNR of 28.2 dB, an SSIM of 0.84 and an LPIPS of 0.15. In plain terms, rendered pixels are off by about 4% of full brightness on average, on frames the model never saw. A simulator has to render the street from viewpoints that were not recorded. When a simulated car changes lanes, all its cameras move sideways together, so the images must be rendered from viewpoints not in the training data. Visually, lane changes of a couple of meters hold up. The street, the parked cars and the buildings stay in place. At a 3-meter lane change, the parked cars on the left end up about a meter from the cameras, closer than any camera ever saw them during training, and they smear (Figure 2, bottom left). A 3DGS implementation on a real street scene, on a 4 GB laptop GPU, with no CUDA kernel: the reconstruction matches what *gsplat* produces and holds up for lane changes of a couple of meters; training takes two hours instead of *gsplat*’s 23 minutes. The code for the entire pipeline, from rendering to training, is on [GitHub](https://github.com/tamoghna21/3DGS_AutonomousDriving). To reproduce the main results: ``` conda env create -f env/environment.yml && conda activate pandaset-splatpython scripts/01_prepare_scene.py --seq 028 # images, masks, LiDAR map and depthpython scripts/03_train.py # the model python scripts/04_replay.py # held-out metrics and the replay videopython scripts/05_novel_views.py # lane-change figurespython scripts/08_eval_offpath.py --ckpts outputs/train/ckpt.pt --labels main # lane-change realism ``` The comparison with *gsplat* needs *gsplat* installed separately. [1] B. Kerbl, G. Kopanas, T. Leimkühler and G. Drettakis, 3D Gaussian Splatting for Real-Time Radiance Field Rendering (2023), ACM Transactions on Graphics (SIGGRAPH 2023) [2] P. Xiao et al., PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving (2021), IEEE ITSC 2021 [3] V. Ye et al., gsplat: An Open-Source Library for Gaussian Splatting (2025), Journal of Machine Learning Research [4] Y. Yan, H. Lin, C. Zhou et al., Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting (2024), ECCV 2024 [5] G. Hess, C. Lindström, M. Fatemi, C. Petersson and L. Svensson, SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving (2025), CVPR 2025 [Gaussian Splatting for Self-Driving Simulation, Built in Plain PyTorch on a Laptop GPU](https://pub.towardsai.net/gaussian-splatting-for-self-driving-simulation-built-in-plain-pytorch-on-a-laptop-gpu-49ed66a0d7c6) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.