Models & learning
Reward model
A reward model is a learned function that estimates task progress, success, preference, or another reward signal from robot observations or trajectories. It can replace or supplement a hand-written reward when training or evaluating a policy, but its output is only as reliable as its labels, coverage, and resistance to shortcuts.
Also known as: learned reward model, robot reward model
Updated
Turning an outcome into a training signal
A reward model watches part or all of an attempt and produces a signal that can be used to compare behaviour. Depending on its design, the output may represent incremental progress, final task success, a scalar reward, or a preference between two trajectories. Inputs can include video, language instructions, robot state, and action history.
This is useful when a task is easier to recognise than to specify as a hand-written formula. A model may learn that a drawer is progressively opening or that one folding attempt is better than another without an engineer assigning a precise reward to every intermediate state.
How robotic reward models are trained
Robometer is one example of a general-purpose video-language reward model. It predicts frame-level task progress and success, and it compares pairs of trajectories. Its training data includes expert, suboptimal, and failed robot attempts, allowing global comparisons to supplement dense progress labels.
Other systems can use a general vision-language model directly as an evaluator. The LeRobot 0.6 release places several approaches behind a shared reward-model interface, including trained classifiers and a zero-shot method that scores whether a trajectory video satisfies its language instruction.
A learned judge can be wrong
A reward model does not establish ground truth merely because it emits a number. It can mistake visual resemblance for completion, miss contact hidden from the camera, or exploit biases in its training tasks. A policy trained with reinforcement learning may then optimise the evaluator's error rather than the intended physical outcome.
Evaluation should therefore test the model on failed, ambiguous, and out-of-distribution trajectories, not only successful demonstrations. High-risk tasks may also require force, state, safety, or human checks alongside visual reward. Real-robot validation remains necessary when a learned reward is used inside a world model or policy-improvement loop.
Sources
Related terms
Models & learning
Reinforcement learning
Reinforcement learning is a method in which an agent learns a policy by interacting with an environment and optimising cumulative reward. In humanoid robotics, actions change the robot and world, while observations, rewards and episode endings provide experience for improving balance, locomotion or manipulation behaviour.
Models & learning
Policy
A policy is the decision rule that maps a robot’s current observations or estimated state, and sometimes a task instruction, to an action or probability distribution over actions. It can be hand-designed or learned from demonstrations, rewards or both. In humanoid robotics, its outputs may be joint targets, torques, end-effector changes or higher-level skills.
Data & collection
Trajectory
A trajectory is a time-ordered sequence of states or observations, actions and, where applicable, rewards generated as an agent or robot evolves. A complete episode or policy rollout often yields a trajectory, but the terms are not universally identical: trajectories may be partial, while episodes have dataset- or environment-defined boundaries.
Models & learning
World model
A world model is a learned predictive model of how an environment changes, often conditioned on a robot’s current state and candidate actions. It may forecast pixels, latent states, rewards or other task-relevant quantities. A policy or planner can use those predictions to compare possible futures before commanding the physical robot.