Simulation & transfer
Synthetic data
Synthetic data is training or evaluation data produced computationally rather than measured directly from the target real-world event. In robotics it can include rendered sensor observations, automatically generated labels, reconstructed or generated scenes, and simulated trajectories. A simulated rollout is one subtype: a temporally linked trajectory produced by executing a controller or policy in a simulator.
Also known as: synthetically generated data
Updated
What counts as synthetic
Synthetic data is an origin category, not one file format. A generator may render RGB, depth, segmentation, optical flow or event-camera observations; expose simulator-native object poses and contact states; compose scenes; generate rare events; or produce complete robot episodes. The Isaac Sim synthetic-data tools separate perception, action-and-event, grasping, mobility and teleoperation workflows, illustrating that the category extends beyond images.
Data can also be generated from reconstructed real scenes or replayed trajectories. Its synthetic status describes how the resulting samples were produced, even when real measurements supplied the assets, initial conditions or motion.
A simulated rollout is narrower
A simulated rollout is a time-ordered interaction: a policy, planner, scripted controller or operator issues actions, the simulator advances, and observations, states, rewards or contacts are recorded. It therefore preserves action-outcome dependencies across an episode.
Synthetic data need not have that structure. A set of independently rendered camera views with segmentation labels is synthetic data but not a rollout. Conversely, one rollout can be replayed to create several synthetic sensor datasets. MobilityGen records robot poses and joint state, then can replay the trajectory to render RGB, depth, segmentation and normals without rerunning physics.
Provenance and limitations
Useful records identify the simulator or generator, asset sources, controller, randomisation ranges, sensor models, annotations and whether a sequence came from live simulation or replay. Simulator-native labels can be precise relative to the virtual scene while still being wrong for reality.
Synthetic coverage can reduce collection cost and include unsafe or rare conditions, but visual realism does not establish physical validity. Models and policies still require evaluation against representative real data and hardware.
Sources
Related terms
Simulation & transfer
Domain randomisation
Domain randomisation is a simulation-training technique that varies selected visual, physical or sensor parameters across generated examples or episodes. By exposing a model or policy to a distribution of environments rather than one calibrated scene, it aims to make real conditions fall within the learned variation and improve sim-to-real transfer.
Simulation & transfer
Sim-to-real
Sim-to-real is the transfer of a model, policy or behaviour developed wholly or partly in simulation to a physical robot or real environment. The central problem is the reality gap: errors in simulated dynamics, sensing, appearance and timing can make a strategy successful in simulation but unreliable or unsafe on hardware.
Models & learning
World model
A world model is a learned predictive model of how an environment changes, often conditioned on a robot’s current state and candidate actions. It may forecast pixels, latent states, rewards or other task-relevant quantities. A policy or planner can use those predictions to compare possible futures before commanding the physical robot.
Data & collection
Trajectory
A trajectory is a time-ordered sequence of states or observations, actions and, where applicable, rewards generated as an agent or robot evolves. A complete episode or policy rollout often yields a trajectory, but the terms are not universally identical: trajectories may be partial, while episodes have dataset- or environment-defined boundaries.