Simulation & transfer
Novel-view synthesis
Novel-view synthesis generates an image of a scene from a camera viewpoint that was not captured directly, using calibrated source observations and a geometric, image-based, or learned scene representation. The output predicts appearance from the requested view; it is not an independent camera measurement or proof that the reconstructed scene has correct physics.
Also known as: NVS, view synthesis, novel-view rendering
Updated
A requested camera pose defines the view
Novel-view synthesis starts with source images and their camera geometry, then predicts what a camera at a different position and orientation would see. The target may lie between captured views or move along a new trajectory. Correct camera intrinsics, extrinsics, scale, and coordinate frames are therefore part of the input contract.
The scene representation can take several forms. RGB-D reprojection moves observed colour and depth into the target camera. Meshes and point clouds can be rendered conventionally. Learned methods such as NeRF optimise a radiance field from calibrated images, while 3D Gaussian splatting renders an explicit set of spatial primitives.
Unseen pixels have to be predicted or left empty
A source view cannot contain surfaces hidden behind a person, robot, or object. Reprojection therefore creates holes and disocclusions when the virtual camera moves. A pipeline may fill them from other observations, a scene model, or image inpainting.
Those pixels are synthesized evidence. A visually plausible fill does not establish the object's true geometry, text, contact state, or material. Image metrics and human inspection can evaluate appearance, but robot collision and manipulation claims need separate geometric and physical validation.
KODA aligns a generated robot view with motion
KODA uses source RGB-D, camera poses, and a generated whole-body trajectory to create the onboard images a Unitree G1 would approximately observe. It removes the human arms, reprojects the scene into the robot camera, renders the robot, composites the layers by depth, and inpaints missing regions.
The important alignment is temporal as well as spatial: the target camera pose comes from the same canonical robot configuration stored beside the image. A fixed camera offset would not follow body motion accurately when the generated robot bends or turns away from the person's path.
Provenance keeps synthesis from becoming measurement
A reusable record should preserve the source frames, sensed or estimated depth, calibration, target camera trajectory, scene representation, renderer, occlusion masks, robot model, compositing order, inpainting method, timestamps, and any quality filters. It should also distinguish observed pixels from reprojected, rendered, and inpainted regions where possible.
Novel-view synthesis can expand viewpoint coverage and align policy observations with a robot trajectory. It cannot recover every unseen surface, guarantee metric accuracy outside the captured volume, or supply contact physics that was absent from the source observations.
Sources
Related terms
Simulation & transfer
3D Gaussian splatting
3D Gaussian splatting is an explicit scene-representation and rendering technique that models a captured scene as optimised anisotropic 3D Gaussian primitives, then projects and blends those primitives to synthesise new camera views. It can reproduce appearance efficiently, but it does not by itself provide watertight collision geometry or physical properties for robot simulation.
Data & collection
Depth data
Depth data records the distance associated with image locations or sensor rays, usually as a depth image in which each pixel stores a metric value relative to a camera. The exact geometry, units, invalid-value convention, and coordinate frame depend on the sensor and encoding. RGB-D data pairs depth with colour imagery; a point cloud is a separate 3D representation derived from or aligned with such measurements.
Hardware & control
Coordinate frame
A coordinate frame is a defined origin and set of oriented axes used to express positions, orientations, motions, forces, or other spatial quantities. A value has no complete geometric meaning until its frame and convention are known. Transformations relate measurements expressed in frames such as world, robot base, camera, end effector, object, or sensor.
Simulation & transfer
Synthetic data
Synthetic data is training or evaluation data produced computationally rather than measured directly from the target real-world event. In robotics it can include rendered sensor observations, automatically generated labels, reconstructed or generated scenes, and simulated trajectories. A simulated rollout is one subtype: a temporally linked trajectory produced by executing a controller or policy in a simulator.
Simulation & transfer
Digital twin
A digital twin is a fit-for-purpose digital representation of a specific physical robot, asset or process that is kept synchronised with its real counterpart through operational data. It may contain geometry, dynamics and simulation models, but the maintained link to an identified real system distinguishes it from an ordinary, standalone simulator.