Simulation & transfer
Real-to-sim
Real-to-sim is the process of using observations or measurements from a real robot, object or environment to construct or update a simulation-ready representation. Depending on the application, this may recover geometry, articulation, calibration and physical parameters, or translate real demonstrations into simulator state, usually to support later analysis, training or sim-to-real deployment.
Also known as: real2sim, real-to-simulation transfer
Updated
The direction of transfer
Real-to-sim starts with evidence from the physical world and turns it into something a simulator can use. Sim-to-real runs in the opposite direction: it takes a model or behaviour developed in simulation and deploys it in reality. A pipeline may use both directions, but they are separate steps with different error sources.
The term is used broadly rather than for one standardised method. Inputs can include RGB or depth images, video, robot joint and torque logs, force measurements, calibration targets and recorded demonstrations.
What the reconstruction contains
A simulation-ready scene needs more than a photorealistic view. Robot interaction may require visual geometry, collision geometry, object separation, joints, coordinate alignment, mass, inertia, friction and controller parameters. Which properties matter depends on the task.
Scalable Real2Sim, for example, combines image-based object reconstruction with robot measurements to estimate collision geometry and inertial properties. RialTo reconstructs real scenes, adds articulation and physical properties, and transfers real demonstrations into simulator state for policy fine-tuning.
Relationship to digital twins
Real-to-sim can create the initial digital representation used by a digital twin, but a reconstructed scene is not automatically a twin. A twin also requires a maintained relationship and synchronisation with a specific physical counterpart.
In a real-to-sim-to-real workflow, the reconstructed simulator becomes a training or testing environment before a policy is transferred back to hardware. Errors can enter at every stage: missing geometry, incorrect contacts, uncertain physical parameters or imperfect demonstration alignment. Physical validation remains necessary even when the virtual scene looks convincing.
Sources
Related terms
Simulation & transfer
Digital twin
A digital twin is a fit-for-purpose digital representation of a specific physical robot, asset or process that is kept synchronised with its real counterpart through operational data. It may contain geometry, dynamics and simulation models, but the maintained link to an identified real system distinguishes it from an ordinary, standalone simulator.
Simulation & transfer
Sim-to-real
Sim-to-real is the transfer of a model, policy or behaviour developed wholly or partly in simulation to a physical robot or real environment. The central problem is the reality gap: errors in simulated dynamics, sensing, appearance and timing can make a strategy successful in simulation but unreliable or unsafe on hardware.
Simulation & transfer
Synthetic data
Synthetic data is training or evaluation data produced computationally rather than measured directly from the target real-world event. In robotics it can include rendered sensor observations, automatically generated labels, reconstructed or generated scenes, and simulated trajectories. A simulated rollout is one subtype: a temporally linked trajectory produced by executing a controller or policy in a simulator.
Data & collection
Trajectory
A trajectory is a time-ordered sequence of states or observations, actions and, where applicable, rewards generated as an agent or robot evolves. A complete episode or policy rollout often yields a trajectory, but the terms are not universally identical: trajectories may be partial, while episodes have dataset- or environment-defined boundaries.