Data & collection
Depth data
Depth data records the distance associated with image locations or sensor rays, usually as a depth image in which each pixel stores a metric value relative to a camera. The exact geometry, units, invalid-value convention, and coordinate frame depend on the sensor and encoding. RGB-D data pairs depth with colour imagery; a point cloud is a separate 3D representation derived from or aligned with such measurements.
Also known as: depth image, depth map, depth images
Updated
A depth image adds geometry to pixels
ROS REP 118 defines a depth image whose pixels store distance along the camera's z-axis and whose camera information supplies the geometry needed for 3D projection. Other systems may report radial range, disparity or a device-specific integer scale, so “depth” is not enough to interpret a file.
Depth can come from stereo matching, structured light, time-of-flight, lidar projection or simulation. Each method has different missing-data patterns, noise, range and behaviour around reflective, transparent or thin objects.
Encoding and calibration determine the metric value
ROS commonly represents depth as 32-bit floating-point metres or 16-bit unsigned integers in millimetres. Intel RealSense devices store 16-bit values multiplied by a camera-specific depth scale. RealSense documentation warns that zero can represent an invalid measurement rather than an object at the camera origin.
Intrinsics are required to deproject pixels into 3D rays. Extrinsic calibration is required to align depth with a separate colour camera or robot frame. An RGB image and depth image with matching dimensions are not necessarily registered.
Depth is useful only with its measurement context
Depth can support object pose, obstacle geometry, scene reconstruction and contact approach. HOI4D combines egocentric RGB-D with hand, object and scene annotations for human-object interaction research.
A dataset should preserve units, encoding, intrinsics, depth scale, minimum and maximum range, invalid-value convention, registration method, frame identifier, timestamps and sensor model. Filtering or inpainting should remain documented so generated values are not mistaken for direct measurements.
Sources
Related terms
Data & collection
Sensor calibration
Sensor calibration is the estimation and documentation of parameters that map raw sensor readings into physically meaningful values and known spatial relationships. In robotics it can include scale, bias, distortion, intrinsic camera parameters, sensor-to-sensor or sensor-to-robot transforms, and timing offsets. Calibration does not remove all noise or drift.
Hardware & control
Coordinate frame
A coordinate frame is a defined origin and set of oriented axes used to express positions, orientations, motions, forces, or other spatial quantities. A value has no complete geometric meaning until its frame and convention are known. Transformations relate measurements expressed in frames such as world, robot base, camera, end effector, object, or sensor.
Models & learning
Pose estimation
Pose estimation is the process of inferring the position and orientation of a body, object, camera, hand, or robot relative to a specified coordinate frame. In three-dimensional robotics this is often called 6D or 6-DoF pose estimation because the result has three translational and three rotational degrees of freedom, even when orientation is stored with more than three numbers.
Data & collection
Egocentric data
Egocentric data is sensor data recorded from the viewpoint of the person or robot performing an activity, most commonly with a head- or body-mounted camera. It can also include audio, gaze, depth or inertial signals. For humanoid learning, it shows hands, objects and actions from an actor-centred perspective.
Data & collection
Robot training data
Robot training data is recorded experience used to train, fine-tune, or adapt models for robot perception, prediction, planning, or control. It can include sensor observations, robot state, actions, task instructions, rewards or outcomes, demonstrations, failures, and embodiment metadata. Not every dataset contains every field, but their timing and physical meaning must be clear.