Data & collection
Robot training data
Robot training data is recorded experience used to train, fine-tune, or adapt models for robot perception, prediction, planning, or control. It can include sensor observations, robot state, actions, task instructions, rewards or outcomes, demonstrations, failures, and embodiment metadata. Not every dataset contains every field, but their timing and physical meaning must be clear.
Also known as: robotics training data, robot learning data, robot training dataset, robot training datasets
Updated
Recorded experience becomes training data through use
Robot training data is an umbrella term rather than one standard file format. A collection may train visual perception, an action policy, a world model, a reward model or another component. Human video, simulation and real robot logs can all contribute, but they provide different levels of action and physical grounding.
The central structure is usually sequential. The RLDS format represents datasets as episodes made of steps and provides fields for observations, actions, rewards and episode boundaries, with room for custom metadata. Other formats organise files differently while preserving the same relationships.
Observation-action alignment is the core requirement
For policy learning, a camera frame or state should be paired with the action taken from that situation and the result that followed. Timing errors can teach the policy to associate an observation with the wrong command. Ambiguous units, frames or action conventions can make a technically loadable dataset physically unusable.
Open X-Embodiment demonstrates both the value and difficulty of combining robot datasets. A common episode format supports joint training, while different cameras, coordinate frames and action semantics still require embodiment-aware processing.
Volume does not replace provenance
DROID paired large-scale real robot demonstrations with a shared hardware setup, synchronised camera streams, calibration and language instructions. Those details make the trajectories interpretable beyond their count.
A useful training package should document collection method, robot and sensors, task and environment, action space, control rate, calibration, quality filters, success and failure treatment, rights and licence. Our guide to humanoid robot training data explains how those fields change the value of motion, vision, teleoperation and interaction records.
Sources
Related terms
Data & collection
Trajectory
A trajectory is a time-ordered sequence of states or observations, actions and, where applicable, rewards generated as an agent or robot evolves. A complete episode or policy rollout often yields a trajectory, but the terms are not universally identical: trajectories may be partial, while episodes have dataset- or environment-defined boundaries.
Data & collection
Demonstration
A demonstration is a recorded example of how an intended task or behaviour is performed, usually represented as a time-aligned sequence of observations, states and actions. For humanoid robot learning, demonstrations may come from teleoperation, kinaesthetic guidance, motion capture or autonomous experts and provide targets for imitation.
Data & collection
Teleoperation
Teleoperation is real-time human control of a robot from a remote or mediated interface. For humanoid training-data collection, the operator’s inputs are mapped to robot motions while cameras, proprioception, commands and outcomes are recorded, producing embodied demonstrations in the robot’s own observation and action spaces.
Simulation & transfer
Synthetic data
Synthetic data is training or evaluation data produced computationally rather than measured directly from the target real-world event. In robotics it can include rendered sensor observations, automatically generated labels, reconstructed or generated scenes, and simulated trajectories. A simulated rollout is one subtype: a temporally linked trajectory produced by executing a controller or policy in a simulator.
Data & collection
Cross-embodiment data
Cross-embodiment data is robot training data drawn from multiple physical embodiments, such as arms, mobile manipulators, quadrupeds or humanoids with different kinematics, sensors and action spaces. The datasets are aligned or packaged so models can learn jointly from experience produced by different robots, although shared formatting does not make their observations or controls physically equivalent.