Simulation & transfer
3D Gaussian splatting
3D Gaussian splatting is an explicit scene-representation and rendering technique that models a captured scene as optimised anisotropic 3D Gaussian primitives, then projects and blends those primitives to synthesise new camera views. It can reproduce appearance efficiently, but it does not by itself provide watertight collision geometry or physical properties for robot simulation.
Also known as: 3DGS, Gaussian splatting, 3D Gaussian splats
Updated
How the representation works
The original 3D Gaussian Splatting paper starts from sparse points produced during camera calibration and optimises a set of three-dimensional Gaussian primitives. Each primitive has a position, anisotropic covariance, opacity, and view-dependent appearance. A visibility-aware rasteriser projects the Gaussians into an image and blends their contributions, allowing new views to be rendered without querying a large neural network at every sample point.
“Splatting” refers to projecting each spatial primitive onto the image plane and accumulating its footprint. The representation is explicit: the scene is stored as many optimised primitives rather than only as weights inside a conventional neural renderer.
Why robotics uses it
A robot policy may need camera observations from viewpoints that were not captured exactly. 3D Gaussian splatting can reproduce the visual appearance of a recorded room from nearby camera poses quickly enough for interactive rendering. That makes it useful in real-to-sim systems, policy evaluation, synthetic observation generation, and scene-specific adaptation.
Splat2Act uses a depth- and normal-regularised Gaussian representation for photorealistic policy observations in building-scale captured scenes. The visual model is paired with a separately extracted mesh for navigation, support-surface discovery, and collision checks.
Appearance is not physics
A Gaussian representation can look geometrically convincing while remaining unsuitable for contact. It does not inherently define watertight surfaces, articulation, mass, friction, or stable collision shapes. A robotics pipeline therefore needs to state which representation drives rendering and which drives physics.
The distinction also affects validation. Novel-view image quality does not prove that an object rests at the correct height, that a robot can pass through a doorway, or that a gripper contact is simulated accurately. Those claims require geometric and physical tests in addition to visual comparisons.
What to retain with the data
A reusable robotics asset should keep the source images or video, camera intrinsics and extrinsics, depth inputs, preprocessing and optimisation settings, representation checkpoint, scale, coordinate frame, and any mesh-extraction step. If the Gaussian scene and collision scene diverge, the package should record their alignment and the measurements used to validate it.
Sources
Related terms
Simulation & transfer
Real-to-sim
Real-to-sim is the process of using observations or measurements from a real robot, object or environment to construct or update a simulation-ready representation. Depending on the application, this may recover geometry, articulation, calibration and physical parameters, or translate real demonstrations into simulator state, usually to support later analysis, training or sim-to-real deployment.
Simulation & transfer
Digital twin
A digital twin is a fit-for-purpose digital representation of a specific physical robot, asset or process that is kept synchronised with its real counterpart through operational data. It may contain geometry, dynamics and simulation models, but the maintained link to an identified real system distinguishes it from an ordinary, standalone simulator.
Data & collection
Depth data
Depth data records the distance associated with image locations or sensor rays, usually as a depth image in which each pixel stores a metric value relative to a camera. The exact geometry, units, invalid-value convention, and coordinate frame depend on the sensor and encoding. RGB-D data pairs depth with colour imagery; a point cloud is a separate 3D representation derived from or aligned with such measurements.
Simulation & transfer
Synthetic data
Synthetic data is training or evaluation data produced computationally rather than measured directly from the target real-world event. In robotics it can include rendered sensor observations, automatically generated labels, reconstructed or generated scenes, and simulated trajectories. A simulated rollout is one subtype: a temporally linked trajectory produced by executing a controller or policy in a simulator.