Data & collection
Egocentric data
Egocentric data is sensor data recorded from the viewpoint of the person or robot performing an activity, most commonly with a head- or body-mounted camera. It can also include audio, gaze, depth or inertial signals. For humanoid learning, it shows hands, objects and actions from an actor-centred perspective.
Also known as: first-person data, egocentric vision, first-person vision
Updated
The actor’s viewpoint
An egocentric camera moves with its wearer, so the recording follows what the person or robot can see while acting. Hands and manipulated objects often occupy much of the frame, while head motion produces viewpoint changes, blur and occlusion. The camera direction is not necessarily the wearer’s gaze unless eye tracking is recorded separately.
Egocentric datasets can extend beyond RGB video. Ego4D includes audio and, for portions of the collection, gaze, stereo views and other modalities. EPIC-KITCHENS records unscripted first-person activity in wearers’ kitchens and adds action narrations and labels. Such annotations turn raw footage into data for recognising actions, anticipating interactions and learning object affordances.
Value and limits for humanoid training
First-person human recordings can show how tasks unfold at human worktops, through doorways and around tools. Their viewpoint may resemble a humanoid’s head camera more closely than a fixed external camera, making them useful for visual pretraining, task understanding and identifying hand–object interactions.
They are not robot demonstrations by default. Human video usually lacks robot joint state, executable actions, force measurements and the robot’s exact camera geometry. Human hands and a humanoid end-effector also have different capabilities. Training a control policy therefore requires alignment, annotation, retargeting or paired robot data. Ego-Exo4D illustrates how synchronised first- and third-person views, camera poses, gaze and inertial signals can add geometric context, but even this richer sensing does not supply humanoid motor commands automatically.
Sources
Related terms
Data & collection
Demonstration
A demonstration is a recorded example of how an intended task or behaviour is performed, usually represented as a time-aligned sequence of observations, states and actions. For humanoid robot learning, demonstrations may come from teleoperation, kinaesthetic guidance, motion capture or autonomous experts and provide targets for imitation.
Data & collection
Teleoperation
Teleoperation is real-time human control of a robot from a remote or mediated interface. For humanoid training-data collection, the operator’s inputs are mapped to robot motions while cameras, proprioception, commands and outcomes are recorded, producing embodied demonstrations in the robot’s own observation and action spaces.
Data & collection
Cross-embodiment data
Cross-embodiment data is robot training data drawn from multiple physical embodiments, such as arms, mobile manipulators, quadrupeds or humanoids with different kinematics, sensors and action spaces. The datasets are aligned or packaged so models can learn jointly from experience produced by different robots, although shared formatting does not make their observations or controls physically equivalent.
Data & collection
Motion capture
Motion capture is the measurement and reconstruction of a person’s or object’s movement over time, commonly as joint positions, orientations or a fitted body model. Optical markers, cameras and inertial sensors can supply the measurements. Humanoid robotics uses the resulting motion sequences for analysis, imitation and retargeting to a robot body.