Models & learning
Reinforcement learning
Reinforcement learning is a method in which an agent learns a policy by interacting with an environment and optimising cumulative reward. In humanoid robotics, actions change the robot and world, while observations, rewards and episode endings provide experience for improving balance, locomotion or manipulation behaviour.
Also known as: RL
Updated
Learning from interaction
An agent observes its situation, selects an action through a policy, receives a new observation and reward, then repeats. The objective is expected cumulative reward rather than accuracy against a fixed action label. Sutton and Barto describe trial-and-error search and delayed reward as defining features: an action may matter because of consequences many steps later.
The resulting training data consists of trajectories containing observations, actions, rewards and boundary signals. Some methods learn while gathering fresh experience; others reuse stored experience. Policy-gradient algorithms update a parameterised policy from sampled trajectories. Proximal Policy Optimization is one widely used formulation that limits how far an update moves the policy.
What it means for humanoids
Humanoid control has large action spaces, coupled joints and strict contact constraints. Reinforcement learning can optimise behaviours that are difficult to specify as individual target actions, including recovery, locomotion and transitions between skills. Reward design remains part of the task specification: an incomplete reward can favour motion that scores well without being safe or useful.
Many physical trials are costly and risky, so training commonly begins in simulation. DeepMimic showed how reinforcement learning could train simulated humanoid characters to track motion-capture clips while satisfying task objectives. A policy intended for hardware still has to tolerate differences in dynamics, sensing and latency. Demonstrations, motion priors and conventional safety controllers can constrain exploration, but they do not remove the need for real-robot validation.
Sources
Related terms
Models & learning
Policy
A policy is the decision rule that maps a robot’s current observations or estimated state, and sometimes a task instruction, to an action or probability distribution over actions. It can be hand-designed or learned from demonstrations, rewards or both. In humanoid robotics, its outputs may be joint targets, torques, end-effector changes or higher-level skills.
Data & collection
Trajectory
A trajectory is a time-ordered sequence of states or observations, actions and, where applicable, rewards generated as an agent or robot evolves. A complete episode or policy rollout often yields a trajectory, but the terms are not universally identical: trajectories may be partial, while episodes have dataset- or environment-defined boundaries.
Data & collection
Motion capture
Motion capture is the measurement and reconstruction of a person’s or object’s movement over time, commonly as joint positions, orientations or a fitted body model. Optical markers, cameras and inertial sensors can supply the measurements. Humanoid robotics uses the resulting motion sequences for analysis, imitation and retargeting to a robot body.
Simulation & transfer
Sim-to-real
Sim-to-real is the transfer of a model, policy or behaviour developed wholly or partly in simulation to a physical robot or real environment. The central problem is the reality gap: errors in simulated dynamics, sensing, appearance and timing can make a strategy successful in simulation but unreliable or unsafe on hardware.