3D Gaussian Splatting

Points as Primitives

In 3DGS, we use a set of points and the corresponding features as primitives:

X = {p_{k}, F_{k}},

where

p_{k} \in R^{3}, F_{k} \in R^{N_{f}} .

In 3DGS, each point has a splat ellipsoid centered at it. Each ellipsoid is defined using a Gaussian and a color

F_{k} = {Σ_{k}, α_{k}, c_{k}}

where

$Σ_{k} \in R^{3 \times 3}$ is the covariance matrix defining the shape of the Gaussian,
$α_{k} \in [0, 1]$ is the opacity,
$c_{k} \in R^{3}$ is the RGB color.

Transform splats into camera coordinate system using camera parameters $C = {R, t}$ .
Sort splats according to depth.
Transform the 3D Gaussian splat distributions into 2D Gaussians in the image space.
Compute final pixel value by accumulating all the splat color values, weighted according to their opacity, visibility and the influence defined by the 2D Gaussian.

The centroid of each splat is projected onto the image plane:

p_{k}^{'} = (R p_{k} + t)_{1 : 2} \in R^{2} .

We also calculate the 2D projection of the covariance matrix:

Σ_{k}^{2D} = (J_{k} Σ_{k}^{'} J_{k}^{T})_{1 : 2, 1 : 2} \in R^{2 \times 2},

where

Σ_{k}^{'} = R Σ_{k} R^{T},

and $J_{k}$ is the affine approximation of the projective transform.

We can finally obtain 2D Gaussian distribution for each splat:

G_{k}^{2D} (x^{'}) = \exp (- \frac{(x^{'} - p_{k}^{'})^{T} Σ_{k}^{2D} (x^{'} - p_{k}^{'})}{2}) .

Given the contribution of each splat on pixel $x^{'}$ , we can accumulate colors by

I_{u v} = c (x^{'}) = \sum_{k = 1}^{K} c_{k} α_{k} G_{k}^{2D} (x^{'}) \prod_{j = 1}^{k - 1} (1 - α_{j} G_{j}^{2D} (x^{'})),

which is quite similar to NeRF, but we replace the opacity with $α_{k} G_{k}^{2D}$ .

Optimal combination between volume rendering and rasterization pipelines. Real-time performance thanks to heavy optimization of the splatting scheme.
Expressive: no limits on surface and scene complexity. Ability to render semi-transparent surfaces.
Intuitive editing supported by modern computer graphics pipelines.