Clustered Error Correction with Grouped 4D Gaussian Splatting

Extending 3D Gaussian Splatting to moving scenes breaks two things at once: densification stops finding the regions that need splats, and a single splat can no longer tell which pixels it matches in the next frame.

3D Gaussian Splatting (3DGS) renders a captured static scene in real time; 4D Gaussian Splatting (4DGS) extends it to scenes that move, and the moving parts are where detail is lost.

Densification fails first. 3DGS adds splats where the viewpoint gradient is large; in 4D that gradient is dominated by temporal transform error and visibility changes and no longer points at the moving regions.

Correspondence fails second. The blue squares crop the same stripes from frame t and frame t+1. Seen whole, the motion is obvious. Seen through one splat, the filled reference mark, the three dashed candidate marks and their candidate transforms are equally plausible, and nothing in the loss objects to the wrong one.

E-D3DGS and STG attack densification with a multi-view DSSIM loss and uniformly sampled error patches. Neither localises the error or tracks which splat belongs to which object, so their renderings flicker and lose fine texture.

Find the errors in the rendered image instead of in the gradient, and give a splat its motion from the group it belongs to instead of from itself.

A lower densification threshold does not help: Baseline Lower Threshold in the ablation below grows Ex4DGS from 426K to 593K splats for no PSNR gain. Instead, compare the rendered frame with the ground truth and go where the pixels are wrong.

A splat projects to an ellipse, so error pixels are clustered into elliptical, single-colour pieces, each the footprint of one splat that is missing or misplaced. A second camera tells which, and the fix is to add a splat or split one.

Correspondence is resolved by the whole pattern rather than the single splat. Splats that move together share one dynamic transform at the group level, each keeping only a static offset. A new splat joins the nearest group and inherits its motion.

CEM-4DGS first trains a Grouped 4D Gaussian Splatting model, then repeatedly clusters its rendering errors into ellipses and corrects each cluster with a new or split splat.

A splat's motion is split in two. The group carries a position and rotation over time, x_G(t) and r_G(t), shared by every member; each splat keeps its own position and rotation relative to that frame, and those never change. The top band applies one rigid motion between Frame t and Frame t+1, so the envelope, the Group Frame and all five members turn together, and the spokes from that frame to each member keep their shape exactly, only rotated. A splat has no per-frame freedom of its own, which is what stops it drifting.

Groups are not given in advance. All splats begin as One Static Group, and a regulariser keeps a splat's displacement d small while it moves with that group, so a d that grows marks a splat that no longer does. An edge joins two of those displaced splats only when they both overlap and point the same way. The three in the middle satisfy both and become a New Dynamic Group; the splat just beneath them overlaps but pulls the other way, and the one off to the left neither overlaps nor agrees. Both keep their d and stay static. This repeats throughout training.

The bottom band is what grouping buys, on the shirt from the Problem section. Hollow marks are four splats at t, filled marks the same four at t+1. Left, Independent Splats: each carries its own transform, and nothing stops it landing a stripe across. Right, Grouped Splats: one shared transform carries all four along the stripes they started on.

The correction cycle runs on the trained model. Grouped 4D Gaussian Splatting renders the left panel, elliptical error clustering takes its difference from the ground truth to give the middle panel, and error-correcting splat addition places a splat in the bracketed region to give the right panel.

Each block shows one frame four times: ground truth, render, the absolute difference between them, brightened so it is visible, and the fix. The marks sit in the same place in all four. In Lacking Splats the brackets enclose a region where the render loses the folds of the fabric because nothing covers them, and After Addition puts them back. In Occlusion the ellipse encloses a hair band that a foreground splat has grown over, and in Split Splats the red inside the ellipse marks the splats that were split, not the annotation.

Which fix applies is decided by a second camera. The ellipse centre is back-projected at its rendered depth into another view; if the ground-truth colour there matches, backproject addition adds a thin-disk splat aligned with the ellipse and joins it to the nearest group, as in Lacking Splats. If it does not, the point is off the true surface, so foreground splitting splits the occluding splat instead, uncovering what was behind it, as in Occlusion.

CEM-4DGS has the highest PSNR on both benchmarks; the margin is larger on Technicolor, where dynamic objects fill more of the frame.

DSSIM₁ and DSSIM₂ follow the two data_range conventions in prior work; other methods' numbers are their official results.

Technicolor Light Field, five scenes at full resolution with one held-out camera: in the PSNR column ours reaches 34.04 dB, 0.39 dB above SWinGS at 33.65, the best prior method. Ours also leads DSSIM₂ and LPIPS.

Neural 3D Video, six scenes at half resolution, Gaussian-splatting rows only: dynamic regions are small and so are the differences. Ours leads the PSNR column at 32.23; STG leads DSSIM, E-D3DGS LPIPS.

The ablation is on the same Technicolor scenes; Baseline is Ex4DGS and each row removes one component. In the PSNR column, Without Group, error correction alone, reaches 33.81 dB; Without Error-based Correction, grouping alone, 33.66; together, 34.04.

The formulation suits rigid or geometrically contiguous deformation, and the correction stage adds training time.

Each splat has a fixed colour and motion is keyframed, so large appearance changes cannot be modelled; translucent objects and volumetric effects such as flames remain hard, as for 3DGS and 4DGS generally. Correction lengthens training on Technicolor from 1.90 h to 3.21 h.