Data & collection
Occlusion
Occlusion occurs when one object, body part, or surface blocks a camera's view of another. In robot learning, occlusion can hide the contact point, object pose, hand configuration, or task outcome that a perception or predictive model needs to infer.
Also known as: visual occlusion, self-occlusion
Updated
The camera loses a line of sight
An object is fully occluded when no relevant part remains visible and partially occluded when only some of it is hidden. Self-occlusion is caused by another part of the same body or object, such as a humanoid arm hiding its torso or a hand covering its own fingertips.
Egocentric manipulation creates a particularly difficult pattern: the hand approaches an object, then covers the exact surface where contact begins. The recording may clearly show the approach and later outcome while hiding the grasp geometry, slip, and object motion in between.
Missing pixels create several plausible states
A tracker can propagate an identity or pose through a short occlusion using earlier frames, motion assumptions, depth, or another camera. That estimate remains uncertain. Meta's Project Aria hand-tracking format stores per-landmark confidence alongside 3D hand positions, which is more useful than presenting every reconstructed joint as equally observed.
For world models, occlusion is also a physics problem. The 30,000-hour egocentric-video study notes that the object geometry most relevant to manipulation is often under the hand, while contacts and forces are not visible at all. A model may predict a plausible hand without enough evidence to determine whether the hidden object should lift, slip, bend, or stay put.
Collection and evaluation should expose uncertainty
Record synchronized views when the task justifies them, preserve calibration and timestamps, and retain confidence or visibility flags for derived tracks. Wrist cameras, external cameras, depth, tactile sensors, and force-torque sensors cover different parts of the missing record; none is a universal substitute for the others.
Evaluate critical moments separately from average visibility. A clip can be clear for most of its duration but hide the grasp, release, or failure that determines its training value. Dataset documentation should state which signals continue through occlusion, how identities are recovered, and when annotations are interpolated rather than directly observed.
Sources
Related terms
Data & collection
Egocentric data
Egocentric data is sensor data recorded from the viewpoint of the person or robot performing an activity, most commonly with a head- or body-mounted camera. It can also include audio, gaze, depth or inertial signals. For humanoid learning, it shows hands, objects and actions from an actor-centred perspective.
Data & collection
Human–object interaction
Human–object interaction is the physical and semantic relationship between a person and an object while the person observes, reaches, grasps, moves, uses, or otherwise acts on it. In robotics datasets, the term often refers to recordings and annotations that connect human body or hand motion with object identity, pose, contact, action, and task context.
Models & learning
Pose estimation
Pose estimation is the process of inferring the position and orientation of a body, object, camera, hand, or robot relative to a specified coordinate frame. In three-dimensional robotics this is often called 6D or 6-DoF pose estimation because the result has three translational and three rotational degrees of freedom, even when orientation is stored with more than three numbers.
Data & collection
Depth data
Depth data records the distance associated with image locations or sensor rays, usually as a depth image in which each pixel stores a metric value relative to a camera. The exact geometry, units, invalid-value convention, and coordinate frame depend on the sensor and encoding. RGB-D data pairs depth with colour imagery; a point cloud is a separate 3D representation derived from or aligned with such measurements.
Hardware & control
Tactile sensing
Tactile sensing is the detection and measurement of physical contact properties at a robot's surface or contact interface. Depending on the sensor, it can report pressure or force distribution, contact location, shear, vibration, slip, texture, temperature, or deformation. Tactile data complements vision by measuring interactions that may be hidden at the point of contact.