mirror of
https://github.com/nerfstudio-project/gsplat.git
synced 2026-10-13 17:42:04 +08:00
Ruilong Li(李瑞龙)
512d366b67
## Summary Training 3DGS with `examples/simple_trainer.py` on 8 B200 GPUs, a good part of the 13.9 ms step was not rendering. Every step waited for a DataLoader worker to decode and undistort one PNG, and then copied it to the GPU as float32. This PR adds an opt-in image cache: each training image is decoded once before training and kept in host memory as uint8, so a step copies a quarter of the bytes and the divide by 255 happens on the GPU. The steady step goes from 13.88 ms to 9.78 ms (**1.42x**) with `--cache_images` on 8 B200 GPUs. Wall time does not improve, because the run's 76 s are dominated by startup, which this PR does not touch; there is a note on that below. I measured one scene on one node. **`--cache_images`**, off by default, COLMAP data only. Upstream decodes and undistorts an image on every access and hands it over as float32. With the flag, `Dataset.preload_images()` decodes every training image on four threads before the `DataLoader` is built, so the workers fork with the cache already in them, and `__getitem__` then takes the cached tensor and keeps it in its decoded dtype. The crop, the depth points and the rest of the sample are still built per access, so `--patch_size` and `--depth_loss` work with the flag on; the loader is untouched, same four workers, same shuffling, same pinning, which is what makes the image order identical by construction. The copy to the GPU is now issued with `non_blocking=True`, and the conversion to float moves after it. The cost is the decoded training set in host memory on every rank, 502 MiB for this scene's 161 images at 1297x840, plus 1.44 s per rank before the loop. It is rejected with a `ValueError` for NCore data, which loads through its own dataset. ### Speedup Result Measured on 8 B200 GPUs, three runs per version, interleaved. The steady step is the mean of loop steps 100 to 874 on rank 0; steps 0 to 99 are excluded because step 0 waits on the slowest rank and refinement starts at step 72. | | unmodified `28e794ca` | this PR, `--cache_images` | change | |---|---|---|---| | training step, steady mean | 13.88 ms | 9.78 ms | **1.42x** | | training step, median | 10.49 ms | 8.61 ms | 1.22x | | run-to-run spread of the steady mean, 3 runs | 0.18 ms | 0.03 ms | | | training step, this PR with the flag off | 13.88 ms | 13.91 ms | unchanged | | training loop, 875 steps | 13.03 to 13.10 s | 9.35 to 11.67 s | | | whole run, launch to exit | 75.7 s | 77.1 s | +1.4 s | | peak GPU memory per rank | 0.95 GB | 0.95 GB | | | host memory per rank | 2.51 GB | 2.98 GB | +0.47 GB, the decoded images | | one-time cost before the loop | | 1.44 s per rank | decoding 161 images on 4 threads | The wide loop time on the right is one run whose step 0 cost 3.50 s against 1.1 to 1.3 s in the others, rank skew at the start of the loop; the steady step is the same in all three. **Wall time does not improve and is slightly worse.** Building the cache costs 1.44 s per rank, which is about what the loop saves over these 875 steps, and the rest of the 76 s is process launch, imports and CUDA/NCCL setup. At 4.1 ms saved per step the cache pays for itself after about 350 steps, so the flag is worth setting on a run longer than that; the default `max_steps` is 30000. The claim here is per-step time, not wall time. With the flag off the step is 13.91 ms against the unmodified 13.88 ms, inside the 0.18 ms spread the unmodified code shows over three runs, so the default path is unchanged. `pytest tests/test_dataset_colmap.py tests/test_strategy.py`: 6 passed with the node's GPUs visible, 4 passed and 2 skipped with `CUDA_VISIBLE_DEVICES=""` (the two skips are the CUDA-only strategy tests, which are pre-existing and untouched here). The new test file builds a small COLMAP dataset, one distorted camera and one not, and asserts that a preloaded item equals the decoded one with and without `patch_size` and `load_depths`, and that preloading leaves the image order and the random crop stream unchanged. `black --check` is clean on the three files. ## Correctness Verification I trained the unmodified code and this branch on the same scene with the same seed and recorded 9016 values per run, 1127 on each of the 8 ranks: the id of the image every one of the 875 steps trained on, and at steps 0 to 9, every 50th step and the last, the loss and its L1 and SSIM parts, the render mean and maximum, the Gaussian counts and the render and alpha shapes. Rather than compare one run against one run, I recorded **8 runs of the unmodified code and 3 of this branch and compared all 55 pairs**. The tolerances were set before the runs, each three times the spread the same value shows over repeats of the unmodified code. 12 of the 24 cross-version pairs pass all 9016 values; the other 12 exceed the tolerances on 1 to 9 of them, so equivalence within these tolerances is not shown pair by pair. **The same comparison inside the unmodified code fails in the same places**: 7 of its 28 unmodified-against-unmodified pairs exceed on 6 values each, and 2 of the 3 branch-against-branch pairs exceed as well. From about step 100 the declared budget sits below this workload's own run-to-run noise, because the rasterizer's backward accumulates gradients atomically and the two runs keep different Gaussians once refinement starts. Measured as distributions instead of pairwise: for every recorded number I took the range the 8 unmodified runs cover and asked how far outside it a run falls, in units of that range's width, judging an unmodified run against the other seven so both versions are treated the same way. **The worst excursion of this branch is 7.69x that width; the worst of the unmodified runs is 29.87x, median 4.8x.** Every step trained on the same image in all 11 runs, and step 0 is bit-identical in all of them. The recording code is not in this PR: one commit adding the calls, applied identically to both sides, on the branches `perf-gsplat-image-cache-verify-base` and `perf-gsplat-image-cache-verify` of my fork `TarzanZhao/gsplat-perf`. | recorded | unmodified vs this PR | tolerance (step 0 / steps 1 to 9 / later) | |---|---|---| | image id of the step, 875 steps x 8 ranks | identical in all 11 runs, so every step trained on the same image on both sides | exact | | step 0, every rank | bit-identical in all 11 runs (rank 7 loss 0.2616417407989502) | exact | | render and alpha shapes | identical in all 11 runs | exact | | loss, L1, SSIM, render mean and maximum | 12 of the 24 pairs within tolerance; the other 12 above it on 1 to 9 values, mid-run and per-rank, where 7 of the 28 unmodified-vs-unmodified pairs are above it too | exact / 3e-4, 2e-3 for the render maximum / 6e-2, 4e-1 for the maximum | | visible Gaussians, Gaussian count after refinement | within tolerance in every pair; of the ten places where the 8 unmodified runs agree and another run does not, all are a visible count off by 2 in about 132000 at an early step, and nine of the ten are unmodified runs disagreeing with each other | exact / 1e-4 / 2e-1 | <details> <summary>Details: hardware, model, full command</summary> **Hardware.** One node, 8 x NVIDIA B200 (sm_100, 183 GB each), 2 x Xeon 8562Y+ (128 threads), driver 580.126.20, CUDA 12.8. Python 3.11.16, torch 2.9.1+cu128, NCCL 2.27.5, gsplat 1.6.0 built from source (`pip install -e . --no-build-isolation`, `TORCH_CUDA_ARCH_LIST=10.0`), fused-ssim @a7c48d6, imageio 2.37.4, pillow 12.3.0, opencv-python-headless 5.0.0.93, pycolmap 4.2.0, torchmetrics 1.8.2, numpy 2.4.6. The scene sits on node-local disk. **Model.** 3D Gaussian Splatting of the Mip-NeRF 360 `garden` scene, 161 training images at 1297 x 840. Gaussians start from the scene's COLMAP points and are sharded across the 8 ranks; SH degree 3, six `torch.optim.Adam(fused=True)` optimizers, fp32 rasterization, `DefaultStrategy` with the refine and reset schedules scaled by `--steps_scaler`, one camera per rank, L1 + SSIM loss. Seed 42 + rank, set by the script. **Measurement.** Against the unmodified `28e794ca`, the script as shipped, three runs per version interleaved (base, PR, base, PR, base, PR), each alone on the node with all 8 GPUs idle beforehand. Both versions export `NCCL_CUMEM_ENABLE=0` and `PYTORCH_NVML_BASED_CUDA_CHECK=1`, which are not part of this PR: the first takes the first step of the unmodified code from 9 to 12 s down to 1 to 5 s on this node, the second is within noise. Step time comes from a `perf_counter` tick at the top of every step on rank 0, recorded by a `sitecustomize.py` outside the repo; the script's own `ellipse_time` agrees with the summed ticks and is the loop row above. The runs that carry the recording calls give the same picture: 13.34 ms and 9.58 ms per step. **Data** (once): `python examples/datasets/download_dataset.py --dataset mipnerf360 --save_dir <data>`, then copy `<data>/360_v2/garden` to local disk. **The measured job**, both versions, with `--cache_images` added on this branch only: ``` export NCCL_CUMEM_ENABLE=0 export PYTORCH_NVML_BASED_CUDA_CHECK=1 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python examples/simple_trainer.py default \ --disable_viewer --data_dir <local>/garden/ --data_factor 4 --result_dir <out> \ --max_steps 7000 --steps_scaler 0.125 \ --eval_steps -1 --save_steps -1 --ply_steps -1 --disable_video --tb_every 0 --port 31200 \ --cache_images # this branch only ``` `--steps_scaler 0.125` is the repo's multi-GPU setting from the `simple_trainer.py` docstring: 875 loop steps of 8 images each, with the refine and reset schedules scaled to match. **The cache and the loader.** `preload_images` runs before the loader starts its workers, so the four workers share the cache of the process they are forked from; the loader, its shuffling and its pinning are untouched, which is why the image order is identical. Where the default start method is spawn rather than fork, each worker would get its own copy of the cache; I measured Linux only. The dataset returns uint8 only after `preload_images`, which nothing but `--cache_images` calls, so other users of `datasets/colmap.py` see float32 as before. Decoding on four threads takes 1.44 s for the 161 images against 5.44 s on one and 0.78 s on eight. **Not exercised.** `--cache_images` with `--patch_size` or `--depth_loss` in a training run (the benchmark scene has neither; both are covered by the tests). Non-8-bit COLMAP images: the cache keeps whatever dtype `imageio` returns and the `/ 255.0` in the trainer is upstream's and unchanged, but only 8-bit data was run. NCore data, which the flag rejects. A scene large enough for the host memory to matter. **Settings tried and rejected** on the unmodified code: `OMP_NUM_THREADS=16`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, `NCCL_NVLS_ENABLE=0` and `NCCL_IB_DISABLE=1` were within noise, and `--packed` was slower. Serving the cache from a zero-worker loader instead of keeping the four workers was 4 to 7 times slower than upstream: collating and pinning the image in the trainer process uses all 128 threads of the node, in 8 ranks at once. </details>
gsplat
gsplat is an open-source library for CUDA accelerated rasterization of gaussians with python bindings. It is inspired by the SIGGRAPH paper 3D Gaussian Splatting for Real-Time Rendering of Radiance Fields, but we’ve made gsplat even faster, more memory efficient, and with a growing list of new features!
News
v1.6.0
Changes on main since the v1.5.3 tag (not yet on PyPI).
- [Aug 2026] Spherical-harmonics updates -- split diffuse and view-dependent operators, fused view-direction computation, and optimized backward kernels.
- [Jul 2026] Sparse 3DGS rasterization -- active-tile rendering with forward/backward support and sparse Gaussian ID, count, and top-contributor queries.
- [Jul 2026] Multi-GPU dense 3DGS -- distributed support for the dense rendering path.
- [Jul 2026] PPISP across densification strategies -- post-processing now works with both the default and MCMC strategies.
- [Jul 2026] Additional performance and code improvements:
- Faster 3DGS and 3DGUT kernels -- optimized dense rasterization, spherical-harmonics feature assembly, and projection across sensor models.
- Orthographic-camera support for 3DGUT.
- Fused SE(3) operations for pose composition and trajectory interpolation.
- More configurable MCMC operations for perturbation and relocation.
- PyTorch 2.7+ is now required, with improved CUDA 12.8 and 13.2 build compatibility.
- [Jun 2026] G-SHARP -- dynamic surgical-scene reconstruction for Gaussian splatting.
- [Jun 2026] New camera & sensor support in the sensors library: pinhole, FTheta, fisheye, and LiDAR models.
- [Jun 2026] Faster rasterization -- a series of performance improvements across the 3DGS and 3DGUT rasterization paths. Noticing roughly 30% improvement on the 3DGUT MCMC path on an NVIDIA A100.
- [Jun 2026] Gaussian ID rasterization -- a new op to rasterize Gaussian IDs, counts, and top contributors per pixel.
- [Jun 2026] Various performance and code improvements:
- Spherical-harmonics improvements -- support for arbitrary channel counts and fp16.
- Profiling and tracing tools for diagnosing performance.
- Robustness and stability fixes across the library.
- Stronger test coverage, plus ahead-of-time (AOT) build/test CI.
- CUDA 13 and NumPy 2 support.
- Simplified package layout -- everything now lives under the
gsplatnamespace.
- [May 2026] Inference Rendering (HiGS) -- An experimental inference-only rendering path based on HiGS (Hierarchically Tiled Gaussian Splatting) is now available under the
experimentalpackage. The inference path uses macro-tile fused rasterization with fp16 scene packing for low-latency rendering of pre-trained Gaussian scenes. For more details, see the Inference Rendering section and the HiGS project page. - [May 2026] Native CUDA MCMC perturb (
inject_noise) speeds up the noise-injection step used in MCMC-style Gaussian optimization. - [Apr 2026] AccuTile adds a conservative ellipse-based tile–Gaussian intersection test on the 3DGS path for tighter work scheduling before rasterization (PR #927).
- [Apr 2026] NCore v4 capture support, including richer camera models and point-cloud loading via
PointCloudsSourceProtocol. See the NCore example. - [Mar 2026] LiDAR rasterization for 3D Gaussian splatting: spinning-lidar camera models,
eval3drendering, depth / hit-distance modes, and related tooling (optional SciPy viapip install "gsplat[lidar]"). - [Mar 2026] TorchScript-oriented deployment: camera models and distortions are also available through PyTorch custom operators and custom classes, not only Python callables.
- [Mar 2026] 3DGUT extensions: external distortion (e.g. windshield-style rigs), optional per-ray inputs with gradients, optional ray-normal outputs, and refactored render modes / extra signals (see 3DGUT notes).
- [Jan 2026] PPISP is integrated as an alternative way of bilateral grid to compensate the training views.
v1.5.3
- [May 2025] Arbitrary batching (over multiple scenes and multiple viewpoints) is supported now!! Checkout the batching guide for more details! Kudos to Junchen Liu.
- [May 2025] Jonathan Stephens makes a great tutorial video for Windows users on how to install gsplat and get start with 3DGUT.
- [April 2025] NVIDIA 3DGUT is now integrated in gsplat! Checkout the 3DGUT integration guide for more details. [NVIDIA Tech Blog] [NVIDIA Sweepstakes]
Installation
Dependence: Please install Pytorch first.
The easiest way is to install from PyPI. In this way it will build the CUDA code on the first run (JIT).
pip install gsplat
Alternatively you can install gsplat from source. In this way it will build the CUDA code during installation.
pip install git+https://github.com/nerfstudio-project/gsplat.git
We also provide pre-compiled wheels for both linux and windows on certain python-torch-CUDA combinations (please check first which versions are supported). Note this way you would have to manually install gsplat's dependencies. For example, to install gsplat for pytorch 2.0 and cuda 11.8 you can run
pip install ninja numpy jaxtyping rich
pip install gsplat --index-url https://docs.gsplat.studio/whl/pt20cu118
To build gsplat from source on Windows, please check this instruction.
Evaluation
This repo comes with a standalone script that reproduces the official Gaussian Splatting with exactly the same performance on PSNR, SSIM, LPIPS, and converged number of Gaussians. Powered by gsplat’s efficient CUDA implementation, the training takes up to 4x less GPU memory with up to 15% less time to finish than the official implementation. Full report can be found in the evaluation results.
python -m pip install -e .
cd examples
python -m pip install -r requirements.txt
# download mipnerf_360 benchmark data
python datasets/download_dataset.py
# run batch evaluation
bash benchmarks/basic.sh
Examples
We provide a set of examples to get you started! Below you can find the details about
the examples (requires installing some extra dependencies via pip install -r examples/requirements.txt --no-build-isolation)
- Train a 3D Gaussian splatting model on a COLMAP capture.
- Fit a 2D image with 3D Gaussians.
- Render a large scene in real-time.
- Train on an NCore v4 capture.
Inference Rendering
gsplat includes an experimental inference-only rendering path based on HiGS (Hierarchically Tiled Gaussian Splatting) in the standalone experimental package, designed for low-latency rendering of pre-trained Gaussian scenes where training gradients are not needed. The inference path packs scene data into compact fp16 layouts and uses a macro-tile fused rasterization pipeline for fast single-camera rendering.
from gsplat.experimental import render_scene, GaussianInferenceScene
The simple_viewer.py example supports the Inference path via the --use_gaussian_render_inference_scene flag. A standalone benchmark comparing Inference rendering against the default rasterization() path is available in examples/benchmarks/gaussian_render_inference_scene/; run gaussian_render_inference_scene_bench.py from the repo root. For more details, see the HiGS project page.
Development and Contribution
This repository was born from the curiosity of people on the Nerfstudio team trying to understand a new rendering technique. We welcome contributions of any kind and are open to feedback, bug-reports, and improvements to help expand the capabilities of this software.
This project is developed by the contributors coming from following institutes (unordered):
- UC Berkeley
- NVIDIA
- ShanghaiTech University
- Amazon
- Meta
- IIIT
- LumaAI
- SpectacularAI
- Aalto University
- CMU
We also have a white paper with about the project with benchmarking and mathematical supplement with conventions and derivations, available on arXiv. If you find this library useful in your projects or papers, please consider citing:
@article{ye2025gsplat,
title={gsplat: An open-source library for Gaussian splatting},
author={Ye, Vickie and Li, Ruilong and Kerr, Justin and Turkulainen, Matias and Yi, Brent and Pan, Zhuoyang and Seiskari, Otto and Ye, Jianbo and Hu, Jeffrey and Tancik, Matthew and Angjoo Kanazawa},
journal={Journal of Machine Learning Research},
volume={26},
number={34},
pages={1--17},
year={2025}
}
We welcome contributions of any kind and are open to feedback, bug-reports, and improvements to help expand the capabilities of this software. Please check docs/DEV.md for more info about development.