Models & learning
Interaction fidelity
Interaction fidelity is the degree to which a learned model reproduces an agent's motion and the physical consequences of that motion on objects and the surrounding scene. It is distinct from visual quality: a prediction can look realistic while preserving the wrong contact, object state, or outcome.
Also known as: object-interaction fidelity, interaction modelling fidelity
Updated
Motion and consequences are separate tests
A predicted interaction contains at least two coupled claims: where the agent moves and how the world responds. A video world model may place a hand accurately while leaving the grasped object behind, or render a plausible object while sending the hand along the wrong path. What 30,000 Hours of Ego-centric Video Does Not Teach therefore reports agent and manipulated-object fidelity separately.
This separation matters in humanoid robotics because a policy, controller, object, and contact can all contribute to the observed outcome. A single task score tells whether the attempt passed; it does not identify which part of the predicted transition was wrong.
Realism is not physical agreement
Perceptual image metrics measure appearance differences. Interaction-fidelity measures instead ask whether task-relevant structures and state changes agree with the recorded outcome. The paper uses Structural Consistency Score: segment and track the hands and objects in predicted and ground-truth clips, then compare their masks over time. Other evaluations may use pose, contact, state, or outcome errors, so “interaction fidelity” does not name one standardised metric.
A convincing video can still have low interaction fidelity when an object disappears after occlusion, deforms like the wrong material, misses a grasp, or remains stationary while the hand moves it. Evaluation should pair visual examples with category-specific metrics and task outcomes rather than substituting one for the others.
State what the evaluation can observe
Report the agent and object definitions, ground-truth source, camera setting, prediction horizon, held-out factors, mask or state estimator, achievable metric bound, and failure categories. For robot rollouts, also state whether the result is generated from demonstrations, policy actions, or measured hardware execution.
Interaction fidelity does not certify safe or dynamically correct robot behaviour. Video cannot directly reveal every contact force, hidden surface, material parameter, or balance state. A strong evaluation localises visible prediction errors and then uses robot state, force, tactile, or simulator evidence for the quantities pixels do not determine.
Sources
Related terms
Models & learning
World model
A world model is a learned predictive model of how an environment changes, often conditioned on a robot’s current state and candidate actions. It may forecast pixels, latent states, rewards or other task-relevant quantities. A policy or planner can use those predictions to compare possible futures before commanding the physical robot.
Data & collection
Human–object interaction
Human–object interaction is the physical and semantic relationship between a person and an object while the person observes, reaches, grasps, moves, uses, or otherwise acts on it. In robotics datasets, the term often refers to recordings and annotations that connect human body or hand motion with object identity, pose, contact, action, and task context.
Models & learning
Object-centric representation
An object-centric representation describes observations, geometry, actions, or trajectories relative to a task-relevant object rather than only in a fixed world frame or robot base frame. In robot learning, it can preserve relationships such as hand-to-object pose and contact while an object is moved, rotated, or transferred between scenes.
Models & learning
Pose estimation
Pose estimation is the process of inferring the position and orientation of a body, object, camera, hand, or robot relative to a specified coordinate frame. In three-dimensional robotics this is often called 6D or 6-DoF pose estimation because the result has three translational and three rotational degrees of freedom, even when orientation is stored with more than three numbers.
Models & learning
Scaling law
A machine-learning scaling law is an empirical relationship between training scale, such as dataset size, model size, or compute, and a measured outcome, commonly model loss. It can guide estimates within the tested setup but does not guarantee similar gains for a different metric or deployment setting.