Models & learning
Markov decision process
A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, transition probabilities and rewards. After an agent chooses an action, the current state and action determine the distribution of the next state and reward. Reinforcement-learning methods use this structure to compare policies by expected cumulative reward.
Also known as: MDP, MDPs, Markov decision processes
Updated
The state–action loop
An MDP specifies a state space, an action space, a transition distribution and a reward function. It also commonly specifies an initial-state distribution, a discount factor and either a finite episode horizon or a continuing task. Sutton and Barto use this framework to define policies, returns and value functions.
The Markov property says that the current state contains the information needed to predict the next-state distribution once the action is known. It does not require deterministic dynamics: the same state and action may lead to several outcomes with different probabilities.
Values and policies
A policy maps the current state to an action or action distribution. The return sums rewards over time, often discounting later rewards. A state-value function estimates expected return from a state, while an action-value function estimates it after first taking a particular action.
Bellman relationships connect those long-horizon values to one reward plus the value of the next state. Value-based, policy-gradient and actor–critic algorithms use that recursive structure in different ways.
When the robot cannot observe the state
A camera frame and joint encoders rarely expose every physical variable in the environment. A partially observable Markov decision process distinguishes hidden state from observation. The Smallwood and Sondik paper formalised control under partial observation.
Robot policies handle partial observability with observation histories, recurrent state, state estimators or belief distributions. Calling the task an MDP is therefore sometimes shorthand for a partially observable deployment in which the learner only receives sensor observations.
Sources
Related terms
Models & learning
Reinforcement learning
Reinforcement learning is a method in which an agent learns a policy by interacting with an environment and optimising cumulative reward. In humanoid robotics, actions change the robot and world, while observations, rewards and episode endings provide experience for improving balance, locomotion or manipulation behaviour.
Models & learning
Policy
A policy is the decision rule that maps a robot’s current observations or estimated state, and sometimes a task instruction, to an action or probability distribution over actions. It can be hand-designed or learned from demonstrations, rewards or both. In humanoid robotics, its outputs may be joint targets, torques, end-effector changes or higher-level skills.
Data & collection
Trajectory
A trajectory is a time-ordered sequence of states or observations, actions and, where applicable, rewards generated as an agent or robot evolves. A complete episode or policy rollout often yields a trajectory, but the terms are not universally identical: trajectories may be partial, while episodes have dataset- or environment-defined boundaries.
Models & learning
World model
A world model is a learned predictive model of how an environment changes, often conditioned on a robot’s current state and candidate actions. It may forecast pixels, latent states, rewards or other task-relevant quantities. A policy or planner can use those predictions to compare possible futures before commanding the physical robot.