Robot Learning Tutorial, Simplified for Programmers

I read the original Robot Learning tutorial and found it valuable, but denser in maths than a practical programmer needs. This article is my programmer-first adaptation of work by Francesco Capuano, Caroline Pascal, Adil Zouitine, Thomas Wolf, and Michel Aractingi. I start with working intuition and use maths only where it clarifies an implementation.

I have changed the language and structure, while retaining the original tutorial's substantive mechanisms, distinctions, caveats, and reported results. The authors also published the original arXiv paper. The Space repository licence publishes its text and original material under Creative Commons Attribution 4.0; it excludes third-party figures, so every caption below retains a source or paper credit. This is an adaptation, not a replacement for their tutorial.

I am keeping the source's scope as well. This is not a complete robotics, deep-learning, reinforcement-learning, diffusion, or flow-matching textbook. It is a route through the ideas that now meet in practical robot learning. I also keep the original authors' sensible position that machine learning can matter greatly without making six decades of classical robotics disposable.

The historical frustration is easy to recognise. The tutorial opens by noting that, more than sixty years after Unimate brought industrial robotics into view, robots still struggle with the unstructured, changing environments people handle routinely. Its central question is whether modern machine learning can close part of that gap without discarding the mechanics and control knowledge that made reliable robots possible.

TL;DR

• I treat a robot policy as an ordinary program with learned parameters: it accepts observations and returns actions.
• Reinforcement learning improves a policy from rewards and interaction. It is powerful, but real hardware makes exploration, reset, simulation fidelity, and reward design expensive.
• Behaviour cloning trains on expert demonstrations instead. It avoids reward engineering, but inherits demonstrator quality and can drift after small mistakes.
• Generative models that use action chunking, such as ACT and Diffusion Policy, represent several valid behaviours and predict short sequences rather than one motor command at a time.
• Asynchronous inference overlaps planning with execution. Generalist vision-language-action models add language, web-pretrained visual knowledge, multi-task data, and specialised action experts.
• I do not read any of this as a reason to throw away classical robotics. Robot kinematics, feedback control, constraints, planners, safety systems, and learned components solve different parts of the same physical problem.

Contents

• LeRobotDataset and collecting usable episodes
• Classical robotics is still the foundation
• Reinforcement learning as a program loop
• Imitation, generative models, ACT, and Diffusion Policy
• Asynchronous inference without idle robots
• Generalist VLAs: pi-zero and SmolVLA

LeRobotDataset and collecting usable episodes

The record every learning method consumes

I find robot learning easiest to understand as a repeated function call:

observation = read_sensors()
action = policy(observation)
next_observation = robot.step(action)

The awkward part is that observation is rarely one tensor. It can contain camera frames, depth, audio, task text, and proprioception: joint positions, velocities, gripper state, and other measurements of the robot's own body. The action may be joint torque, joint velocity, a target configuration, an end-effector command, or a chunk of several future commands. My earlier guide to robot training data modalities explains why those distinctions matter at dataset level.

LeRobotDataset gives these streams one interface. A recorded trajectory, also called an episode, is a time-ordered run of observations and actions, plus task and timing metadata. The format supports physical manipulators such as SO-100/SO-101 and ALOHA-2, humanoid arms and hands, simulated robots, and even autonomous-driving data. The point is not that every embodiment looks alike. The point is that loaders can expose their differences through one schema.

LeRobot covers the surrounding stack too: low-level reads and writes for supported robot hardware, extension points for new platforms, PyTorch implementations of RL and imitation policies, experiment tracking, and an inference path that can separate action planning from action execution.

I would think of the on-disk design as three coordinated stores:

• data/* contains low-dimensional, frame-by-frame fields such as state and action in chunked Parquet files. Multiple episodes are concatenated so millions of episodes do not become millions of tiny files.
• videos/* contains camera streams encoded as MP4. Frames are grouped by camera and chunked across directories for the same file-system reason.
• meta/* is the index that puts everything back together. info.json defines feature names, shapes, dtypes, frame rate, LeRobot version, and path templates. stats.json carries mean, standard deviation, minimum, and maximum values for normalisation. tasks.jsonl maps language instructions to task IDs. episodes/* records episode lengths, tasks, and offsets into the shared Parquet and video files.

That separation is practical software engineering. The API can return an episode-shaped sample while storage stays compact, memory-mappable, and suitable for the Hugging Face Hub.

Windows, batches, and streaming

Most learning code does not want a single frame. An RL agent may need recent observations because one image cannot reveal velocity or hidden state. An imitation model may predict a future action chunk. delta_timestamps asks the dataset for those windows in seconds, not in hard-coded row offsets. Missing frames at an episode boundary are padded and accompanied by a mask.

This is a concise excerpt from the real dataset-loading snippet:

delta_timestamps = {
    "observation.images.wrist_camera": [-0.2, -0.1, 0.0],
}
 
dataset = LeRobotDataset(
    "lerobot/svla_so101_pickplace",
    delta_timestamps=delta_timestamps,
)
 
loader = torch.utils.data.DataLoader(dataset, batch_size=16)
for batch in loader:
    observations = batch["observation.state"].to(device)
    actions = batch["action"].to(device)
    images = batch["observation.images.wrist_camera"].to(device)

The returned image shape has a leading window dimension, so three requested timestamps produce something like [3, channels, height, width] per sample. A PyTorch DataLoader then collates each dictionary key into a batch. Replacing LeRobotDataset with StreamingLeRobotDataset avoids a full local download; the tutorial reports roughly 80–100 iterations per second depending on connectivity, while retaining enough randomisation to avoid training on long, highly correlated episode runs.

I would still benchmark that number on the actual network, codec, workers, and batch shape. It is a reported operating range, not a service-level promise.

Recording with a leader and follower arm

The collection path is equally concrete:

1. Configure the follower robot, leader teleoperator, camera names, resolution, ports, calibration IDs, and frame rate.
2. Derive dataset features from the hardware's observation and action schemas.
3. Create a LeRobotDataset with those features, the robot type, and video settings.
4. Connect the robot and teleoperator, pass the task description into the recording loop, record a timed episode, reset the scene, and either save or re-record it.
5. Push the finished dataset to the Hub only after checking episodes and metadata.

The real recording snippet uses five 60-second episodes at 30 fps, with ten seconds for reset:

dataset = LeRobotDataset.create(
    repo_id=f"{HF_USER}/robot-learning-tutorial-data",
    fps=30,
    features=dataset_features,
    robot_type=robot.name,
    use_videos=True,
    image_writer_threads=4,
)
 
record_loop(
    robot=robot,
    teleop=teleop,
    dataset=dataset,
    fps=30,
    control_time_s=60,
    single_task=TASK_DESCRIPTION,
    display_data=True,
)
dataset.save_episode()
dataset.push_to_hub()

The source also points to the lerobot-record CLI and utilities such as lerobot-find-port. I would inspect camera keys, calibration files, units, and timing before collecting volume. A perfectly trained model cannot repair a wrist camera logged under the wrong key or an action stream shifted by several frames.

There is a source-rendering trap worth stating plainly. The rendered tutorial currently duplicates the same generic dataset-loader placeholder in several code boxes, including collection, HIL-SERL, ACT, Diffusion Policy, asynchronous inference, π0, and SmolVLA. The Space's source files contain the actual implementations. I therefore quote only short, real excerpts here and link every full snippet in its relevant section rather than repeating the broken placeholders. The snippets have their own MIT licence, and LeRobot's API can move, so I would also check the installed version's documentation.

Classical robotics is still the foundation

Analytical and learned models solve different problems

Classical robotics starts from explicit structure: rigid links, joints, geometry, dynamics, contact assumptions, constraints, planners, and feedback controllers. Learning starts from examples or interaction and fits parameters from data. The original tutorial calls these “explicit” dynamics-based and “implicit” learning-based models, which is useful shorthand but not a strict taxonomy. A learned dynamics model is still model-based, and a neural policy is still an explicit executable function. I see analytical versus learned, model-based versus model-free, and modular versus end-to-end as separate axes. A learned perception system can feed a classical planner; model-predictive control can guide temporal-difference learning; a learned contact model can improve an otherwise analytical controller.

Robotics also contains different motion problems. Manipulation changes the environment through the robot. Locomotion changes the robot's position, using wheels or legs. Mobile manipulation combines both and exposes many more control variables. Whole-body control on a humanoid adds balance, contact switching, and coordinated motion, while an industrial arm bolted beside a fixed conveyor faces a narrower problem.

The hardware barrier is falling. A SO-100-class arm costs hundreds of euros and can use 3D-printed parts, while an industrial Franka arm costs tens of thousands. Low cost does not make calibration, safety, or control trivial, but it makes repeated real-world data collection available to far more programmers.

A two-joint arm as executable geometry

The tutorial simplifies a SO-100 from five motion joints plus a gripper to shoulder lift, elbow flex, and a gripper. Ignoring the gripper for motion leaves a two-link planar arm. Each configuration is then just q = [theta1, theta2]. The toy model initially allows each angle to span [-pi, pi], a full turn; the real arm's body, base, wiring, and joint limits make that assumption deliberately unrealistic.

For two links of lengths l1 and l2, forward kinematics is ordinary trigonometry:

x = l1 * cos(theta1) + l2 * cos(theta1 + theta2)
y = l1 * sin(theta1) + l2 * sin(theta1 + theta2)
pose = [x, y]

The source's longer equation also says the hand cannot be farther from the base than l1 + l2; with equal links, the squared distance is at most (2 * l) ** 2. Forward kinematics implements pose = fk(joints). Inverse kinematics asks for the opposite direction: find a valid joints value that places the hand at target_pose.

When there is no neat closed form, the source's inverse-kinematics equation becomes:

best_q = argmin(
    squared_distance(fk(q), target_pose)
    for q in feasible_joint_configurations
)

The word feasible carries most of the pain. A floor restricts the first joint to its upper half-plane and makes the second joint's legal range depend on the first. A shelf adds collision constraints. A real robot also has joint limits, self-collision, cables, torque limits, and objects that move.

Inverse kinematics finds a goal pose, not the route to it. A desired route is a timed list such as target_poses = [p0, p1, ..., pK], supplied by an expert or generated by a motion planner; solving a full optimisation at every waypoint is expensive. Differential IK instead relates a small joint-velocity change to a small hand-velocity change through a Jacobian:

J = jacobian(fk, q)
q_velocity = argmin(
    squared_norm(J @ candidate - desired_pose_velocity)
    for candidate in feasible_joint_velocities
)
# Common closed-form shorthand:
q_velocity = pseudoinverse(J) @ desired_pose_velocity
q_next = q + dt * q_velocity

The Jacobian is simply a runtime sensitivity table: if I move each joint a tiny amount, how does the hand move? The pseudoinverse finds the joint velocity that best matches the requested hand velocity when an exact inverse is unavailable.

Feedback, disturbances, and the engineering bill

Open-loop tracking assumes the model and the world are right. The next figure adds a moving box. If its horizontal speed changes unpredictably, a precomputed path can become a collision.

The source models unpredictable motion as box_velocity += normal_noise(0, 1). A proportional feedback loop then corrects what the model missed:

pose_error = target_pose - measured_pose
corrected_pose_velocity = desired_pose_velocity + kp * pose_error
q_velocity = pseudoinverse(J) @ corrected_pose_velocity

PID, feedback linearisation, LQR, and model-predictive control extend the same idea. They can reject moderate disturbances and stabilise a well-modelled system. The caveat is practical: gains are system-specific, contact modes and constraints can switch abruptly, an undamped pseudoinverse becomes unstable near singular configurations, and conservative tuning may be needed for stability.

This is where I agree most strongly with the original tutorial's balanced position. Classical methods are excellent when geometry is known, environments are controlled, guarantees matter, and constraints must be explicit. Their limitations appear when a pipeline becomes a long chain of bespoke perception, state estimation, mapping, planning, IK, and low-level control modules; when raw RGB, depth, tactile, and audio need hand-built interfaces; when every task needs new goals and constraints; or when friction, deformable objects, and intermittent contact defeat the simplified model.

Learning addresses those scaling problems by fitting perception-to-action mappings from data. It does not repeal geometry or physics. A sensible robot still uses calibrated hardware, constraints, watchdogs, emergency stops, and often classical controllers below or around the learned component.

Reinforcement learning as a program loop

State, action, reward, repeat

RL fits robotics because both are sequential. An arm reaching for a block must revise its command after every observation. A walking robot must keep moving its centre of mass without falling. The useful abstraction is an agent interacting with an environment, not a static predictor processing independent rows.

I keep generalist models beside imitation learning in that map because they are still largely trained to reproduce demonstrated actions. Language conditioning and scale change the scope, but the underlying supervision remains closely related.

The standard formal wrapper is a Markov decision process, or MDP. I translate its symbols into this interface:

class Environment:
    state_space          # every possible physical state
    action_space         # every command the agent may issue
    dynamics             # distribution of next_state given state and action
    reward_fn            # score for a transition
    initial_state_dist   # how episodes begin
    horizon              # maximum episode length
 
gamma = 0.99             # discount applied to later rewards

The true state may contain joint configuration, joint velocity, object poses, and hidden physical variables. The agent often receives only an observation, such as cameras and encoders, so stacking recent observations can reduce partial observability. Formally that is a partially observable MDP: the deployed policy is closer to policy(action | observation_history) than policy(action | hidden_state). Actions can be torques, velocities, joint targets, or Cartesian commands.

dynamics(state, action) returns a distribution because the next state may be uncertain. The Markov assumption says next_state needs only the current state and action, not the entire log, and policy(action | state) needs only the current state. Under that assumption, the chance of a whole episode is:

trajectory_probability =
    initial_state_probability
    * product(
        transition_probability(state[t + 1] | state[t], action[t])
        * policy_probability(action[t] | state[t])
      )

That is the source's trajectory-factorisation equation in programmer form. A finite run is [(s0, a0, r0), (s1, a1, r1), ..., sT]. A finite horizon ends at T; an infinite-horizon formulation does not set such a bound.

A reward function compresses task preference into a number. The tutorial's locomotion example is equivalent to reward = horizontal_position * horizontal_speed - 1 / body_height: move quickly in the desired direction, but penalise low height as the robot falls. I would treat that as an illustration of reward shaping, not a production design. The singular penalty near zero and the multiplication of position by speed can create incentives that deserve testing.

The return adds rewards over the episode while reducing the weight of distant ones:

episode_return = sum(
    gamma**t * reward[t]
    for t in range(horizon)
)
effective_horizon_roughly = 1 / (1 - gamma)

The training objective is maximise mean(episode_return) across trajectories generated by the current policy and fixed environment dynamics. In code, the dynamics affect sampled data but are not parameters the agent updates.

Two cached estimates make that objective usable:

V(state) = expected future return after starting in state
Q(state, action) = expected future return after taking action, then following policy
 
Q(state, action) =
    expected(reward + gamma * V(next_state))
 
V(state) =
    expected(Q(state, action) for action sampled from policy)

These are the source's state-value, state-action-value, and Bellman consistency equations. They say that a long-term estimate today must agree with one immediate reward plus the discounted estimate tomorrow.

From a Q table to SAC

The algorithm family tree looks intimidating, but the progression is mostly the removal of one engineering limitation at a time.

Q-learning starts with a table. If Q_star were known, action selection would be action = argmax_a Q_star(state, a). The rendered source writes max in one place, but the returned object must be the action, so argmax is the programmer-correct operation. Its displayed expected backup is value iteration:

Q_next[s, a] = mean(
    reward + gamma * max(Q_current[next_state, next_action])
    for sampled_next_state in transitions(s, a)
)

Sample-based Q-learning performs the same idea one observed transition at a time:

target = reward + gamma * max(Q[next_state, next_action])
Q[state, action] += learning_rate * (target - Q[state, action])

Under the usual finite/discrete assumptions and enough updates, the table approaches the optimal values. It fails to scale when states are images or actions are continuous motor vectors.

Deep Q-Networks replace the table with Q_network(state, action). The training target uses a previous or target network:

target = reward + gamma * max(
    Q_target(next_state, next_action)
    for next_action in discrete_actions
)
td_error = target - Q_network(state, action)
loss = mean(td_error**2)

A replay buffer stores (state, action, reward, next_state) and samples old transitions out of sequence. That both breaks correlations and allows off-policy learning: the behaviour that collected the data can differ from the policy being improved.

Deterministic Policy Gradient handles continuous actions by adding an actor, action = actor(state), rather than searching every possible real-valued action. Its gradient is just the chain rule through the critic:

actor_update_direction =
    critic_gradient_with_respect_to_action
    * actor_gradient_with_respect_to_parameters

Updating actor parameters in that direction asks the actor to emit actions the critic scores more highly. DDPG combines that actor with deep networks and replay. Its target replaces the impossible max over all continuous actions with Q_target(next_state, actor_target(next_state)).

Soft Actor-Critic, or SAC, keeps a stochastic actor and adds entropy. Its objective is roughly:

soft_objective = mean(
    reward + alpha * policy_entropy(state)
)
soft_target = reward + gamma * (
    Q_target(next_state, sampled_next_action)
    - alpha * log_probability(sampled_next_action)
)

The alpha term rewards a policy for preserving several plausible actions instead of collapsing too early. The policy update fits the next tractable distribution, often a Gaussian, towards one proportional to exp(Q / alpha). This maximum-entropy formulation tends to produce more diverse exploration and has been robust in continuous-control work. Because SAC is off-policy, it can reuse replayed transitions and is generally more sample-efficient in this setting than an on-policy method such as PPO.

These are model-free algorithms: they learn values or actions without using a predictive dynamics model for planning. The MDP still has dynamics, and training inside a simulator does not by itself make an algorithm model-based. LeRobot's TD-MPC sits on the other side of that distinction because it learns a model and uses it for model-predictive control.

Simulation, reality gaps, and reward design

Physical exploration can command unsafe velocity, torque, or self-colliding poses. Episodes need manual resets, hardware wears, and even a good off-policy algorithm may require many transitions. Simulation removes much of the immediate risk and can generate experience faster, but it introduces another model.

Domain randomisation trains each simulated episode with different physics or appearance. In code, the source's dynamics equation is:

random_parameters = sample(randomisation_distribution)
environment = Simulator(
    friction=random_parameters.friction,
    centre_of_mass=random_parameters.centre_of_mass,
    lighting=random_parameters.lighting,
    # ...
)

The aim is to make the real world look like one more sampled domain.

Choosing the randomisation distribution is itself a modelling job. Too narrow and transfer fails. Too wide and the policy is over-regularised and performs poorly everywhere. AutoDR starts with a uniform range and widens fixed bounds as performance improves. DORAEMON instead learns updates to a more flexible Beta distribution, maximising its entropy in an outer loop while maintaining an RL performance constraint. SimOpt-style methods repeatedly interleave simulated training with real rollouts to update the distribution; DROPO instead tunes it against one pre-collected set of real trajectories.

I would not call any of these a complete answer. Contact-rich cloth folding, soft materials, friction, and coupled grasp dynamics can remain too expensive or inaccurate to simulate. The original tutorial makes the stronger point that a perfect randomisation distribution would still not make an impractical simulator practical.

Reward design is the second hard limit. A sparse 1 if shirt_is_folded else 0 is honest but gives little learning signal. A dense score for every intermediate fold is informative but easy to get wrong, game, or optimise into a local shortcut. Some domains, such as tokamak control or stratospheric navigation, still make simulation the only feasible training environment. For many manipulation tasks, however, real demonstrations offer a less contrived source of intent.

Prior data, learned rewards, and HIL-SERL

The path to practical real-world RL is a sequence of useful additions:

• SAC reuses old transitions through an off-policy replay buffer.
• RLPD mixes prior offline data with new online data, sampling each in equal proportion. It starts online learning from scratch rather than requiring a separate pretraining phase, and uses details such as LayerNorm and critic ensembles to reduce value overestimation.
• SERL adds tooling for real robots: a classifier can learn success from positive and negative images, a forward controller performs the task, and a backward controller resets it. Expressing state and action relative to the initial end-effector frame, then randomising that initial pose, provides goal variation without rebuilding the scene every episode.
• HIL-SERL adds human intervention during online learning. Demonstrations seed the offline buffer; autonomous transitions enter the online buffer; corrections enter both, making them more likely to be sampled.

For the learned reward, the source's equation means reward(state) = log(classifier_probability(success | state)). In practice, the LeRobot example trains a ResNet-18-based classifier on positive and negative frames, then freezes it inside the actor.

The tutorial summarises HIL-SERL as reaching more than 99% success on varied manipulation tasks in one to two hours. The paper describes near-perfect success and practical training in roughly one to 2.5 hours. I would carry both as reported experimental results, not a guarantee for a new robot, camera, task, or reward classifier.

The actual LeRobot learner makes the 50/50 RLPD mechanism obvious:

online_batch = online_buffer.sample(batch_size // 2)
offline_batch = offline_buffer.sample(batch_size // 2)
 
batch = {
    key: torch.cat([online_batch[key], offline_batch[key]], dim=0)
    for key in online_batch
    if key in offline_batch
}
 
loss, _ = policy_learner.forward(batch)
loss.backward()
optimizer.step()

The actor can stay on the robot and the learner can run on accelerated hardware, even on another networked machine. Queues carry transitions in one direction and fresh parameters in the other. Human intervention metadata decides whether a transition also enters the demonstration-heavy offline buffer.

The four full source files are worth reading together:

• Train the reward classifier
• Run the actor and detect interventions
• Run the learner and combine replay buffers
• Wire the HIL-SERL processes, robot, teleoperator, and buffers together

I would add hard safety limits outside the learned actor, validate the reward classifier in the exact deployment view, preserve interventions with timestamps, and expect the scene-reset logic to be task-specific. HIL-SERL makes real-world RL more plausible; it does not make unsupervised hardware exploration sensible.

Imitation, generative models, ACT, and Diffusion Policy

Behaviour cloning is supervised control

Imitation learning removes the environment reward from the training set. Instead of transitions such as (state, action, reward, next_state), I train on expert episodes containing (observation, action) pairs. An observation can combine images and robot state; an action is what the expert's leader arm or another operator commanded.

The source's dataset notation translates to:

dataset = [
    [(observation_0, expert_action_0), ..., (observation_T, expert_action_T)],
    # N expert episodes, each allowed to have a different length
]

No reward appears. The simplest cloned controller is a deterministic function predicted_action = model(observation), trained with:

loss = mean(
    action_loss(expert_action, model(observation))
    for observation, expert_action in shuffled_pairs
)

That is the source's supervised risk-minimisation equation. The pairs are not truly independent and identically distributed because they came from sequential episodes. Shuffling weakens the short-range correlation but does not erase the data-collection policy. DAgger addresses distribution shift by repeatedly running the learner, asking an expert what should have happened in the learner's visited states, and aggregating the new labels. The tutorial intentionally keeps to the offline case where no more demonstrations can be collected.

Offline cloning avoids dangerous exploration, reward shaping, online resets, and explicit success instrumentation during training. It also encodes the demonstrator's goal and preferences. The bill arrives elsewhere:

• A cloned policy cannot systematically exceed the quality and coverage of its demonstrations without another learning signal.
• A tiny prediction error can move the robot into a state absent from the dataset. The next prediction is then less reliable, causing compounding error or covariate shift.
• Human demonstrations are multimodal. Going left or right around an obstacle can both be correct. A point regressor trained with mean-squared error may average the two and drive straight into it.

Why latent and generative models help

A generative controller learns a distribution rather than one average answer. Given a scene, it can represent several plausible action sequences and sample one coherent mode.

The first useful mental model is a hidden variable, usually called z. It can encode style, task, route, or another factor the dataset did not label. The source's marginalisation equation means:

probability(observation, action) =
    sum_over_every_possible_z(
        probability(observation, action | z)
        * probability(z)
    )

The mathematical version uses an integral when z is continuous. I read it as dependency injection for unobserved context: the same open-gripper action can be likely while picking and unlikely while pushing.

A Variational Auto-Encoder, or VAE, learns two networks:

• encoder(observation, action) approximates the posterior distribution over z.
• decoder(z) or decoder(z, observation) reconstructs a likely action or observation-action pair.

The source walks through the evidence calculation by inserting approx_posterior / approx_posterior, turning an integral into an expectation, then applying Jensen's inequality. The implementation result is the evidence lower bound, or ELBO. I would not transcribe the rendered log-and-sum derivation into code; the conventional minibatch objective is:

z_dist = encoder(observation, action)       # usually a Gaussian
z = z_dist.rsample()
reconstruction = decoder(z)
 
reconstruction_loss = mse(reconstruction, [observation, action])
regularisation_loss = kl_divergence(z_dist, standard_normal)
loss = reconstruction_loss + beta * regularisation_loss

This covers each part of the source equations:

• Maximising data likelihood becomes minimising negative ELBO.
• The expected decoder log-likelihood becomes reconstruction loss. With an isotropic Gaussian decoder of fixed variance, it reduces to squared error plus a constant.
• The KL term keeps the approximate posterior close to a simple prior, commonly Normal(0, I), so latent samples remain usable.
• Monte Carlo samples estimate the expectation.
• A larger beta weights compression more heavily and makes the latent channel less expressive.

That is enough mathematics to implement and debug a VAE. The full original paper contains the derivation.

Diffusion and flow matching as iterative generators

A VAE uses one latent sample. A diffusion model uses a chain of progressively noisier variables. The source factors their joint probability as:

generation_probability =
    probability(pure_noise)
    * product(
        probability(slightly_cleaner_sample | noisier_sample)
      )

Marginalising every intermediate variable gives the data probability. The forward corruption chain is fixed and Markovian; the reverse chain is learned.

I would not assign literal meanings to these latent levels, but the source offers a useful intuition: higher levels can carry coarse trajectory structure while levels nearer the data recover fine motion adjustments.

With a Gaussian schedule, one corruption step is:

noisy_t = sqrt(1 - beta_t) * less_noisy_previous + sqrt(beta_t) * normal_noise()

Repeated small steps converge towards a standard Gaussian. A useful property is that training can jump directly from clean data to any noise level:

alpha_t = 1 - beta_t
alpha_bar_t = product(alpha_1, ..., alpha_t)
noisy_t = sqrt(alpha_bar_t) * clean + sqrt(1 - alpha_bar_t) * noise

The source's long likelihood derivation again inserts a forward process divided by itself and applies Jensen's inequality. The rendered indices and conditionals are inconsistent, so I would not implement that expression literally. The standard DDPM lower bound splits into three jobs: reconstruct clean data from the first noisy level, make the final noise match the simple prior, and make each learned reverse step match the tractable posterior of the forward process. Gaussian choices make the KL terms closed-form.

The DDPM paper simplifies that into a noise-prediction loss:

t = random_integer(1, num_diffusion_steps)
clean = sample(dataset)
noise = standard_normal_like(clean)
noisy = sqrt(alpha_bar[t]) * clean + sqrt(1 - alpha_bar[t]) * noise
 
predicted_noise = network(noisy, t)
loss = mse(predicted_noise, noise)

That pseudocode is the programmer version of the source's diffusion objective. The network learns the total displacement caused by a randomly long corruption process.

At sampling time, each reverse update has the source's structure:

cleaner = (
    noisy
    - noise_scale(beta_t, alpha_bar_t) * predicted_noise(noisy, t)
) / sqrt(alpha_t)
cleaner += sigma_t * fresh_noise

Running that step from T down to zero turns random noise into a likely sample. The manifold interpretation is optional but helpful: training learns a direction back towards the thin region where real data live; denoising repeatedly projects an off-distribution point towards that region.

Diffusion can require hundreds of expensive steps. Flow matching generalises the idea by learning a continuous vector field that transports a simple distribution into the data distribution. Its core equation is an ODE:

dz_dt = vector_field(time=t, sample=z)
z_at_time_0 = sample(simple_prior)

Integrating that derivative from zero to one generates a sample. The source also shows diffusion as one special field:

diffusion_field =
    schedule_derivative / (1 - schedule**2)
    * (schedule * current_sample - clean_sample)
 
schedule(t) = exp(-0.5 * integral(noise_rate, 0, t))

That expression matters mainly because it proves diffusion fits inside the broader flow framework. Conditional flow matching can use a much simpler straight path:

t = uniform(0, 1)
prior_sample = sample(simple_prior)
data_sample = sample(dataset)
 
point_on_path = (1 - t) * prior_sample + t * data_sample
target_velocity = data_sample - prior_sample
loss = mse(vector_field(point_on_path, t), target_velocity)

At inference, forward Euler is enough to express the source's solver:

for t in timesteps_between_0_and_1:
    z = z + dt * vector_field(z, t)

Straighter optimal-transport-style paths can require fewer steps than a random Brownian route, reducing inference cost while retaining a multimodal generator. The Flow Matching paper is the primary source for that framework.

ACT: a conditional VAE over action chunks

Action Chunking with Transformers, or ACT, tackles both multimodality and compounding error. Instead of predicting action[t], it predicts actions[t:t + chunk_size]. That is closer to how I would plan a short function body than one machine instruction at a time.

ACT learns the conditional distribution probability(action_chunk | observation), not the full joint distribution of observations and actions. Recovering that conditional from a learned joint would require dividing by the integral of the joint across every possible action, which is generally intractable for a neural network. A conditional VAE models the useful distribution directly.

The source first displays the general conditional-VAE form with a learned prior conditioned on the observation. ACT's actual recipe simplifies that prior to Normal(0, I):

posterior = encoder(observation, expert_action_chunk)
latent_prior = standard_normal
z = posterior.rsample()
predicted_chunk = decoder(z, observation)
 
loss = (
    reconstruction_loss(predicted_chunk, expert_action_chunk)
    + beta * kl_divergence(posterior, latent_prior)
)

The general equation's learned prior_network(observation) is therefore absent from the implementation, and evaluation sets z = zeros. The encoder embeds the expert action chunk, positional information, current proprioception, and a learned [CLS] token into z. For efficiency, it does not encode camera images. The encoder is training-only.

The decoder embeds every camera through pretrained visual encoders, adds positional embeddings, combines the visual tokens with proprioception and z, and uses an encoder-decoder Transformer to emit the action chunk. At inference, ACT discards the training encoder and deterministically sets z = zeros, while the live images and robot state still condition the decoder.

The ablations explain why the machinery exists. On scripted demonstrations, a plain L1 = abs(expert_action - predicted_action) objective performed comparably to the generative objective. On human demonstrations, replacing the variational objective with plain supervised learning reduced success by 33.3 percentage points in the reported comparison. Removing action chunking produced 1% success versus 44% with a chunk size of 100. ACT also queries at every timestep and exponentially averages overlapping predicted chunks, reducing the open-loop period.

ALOHA is part of the same contribution: an off-the-shelf, low-cost bimanual teleoperation setup built for fine tasks such as opening a lid and slotting a battery. The paper reports six real-world tasks, 80–90% success, and about ten minutes of demonstrations in its headline experiments. The tutorial's comparison says the two-arm setup cost roughly one single Franka arm.

The real LeRobot training snippet derives feature schemas and windows from dataset metadata:

metadata = LeRobotDatasetMetadata(dataset_id)
features = dataset_to_policy_features(metadata.features)
 
cfg = ACTConfig(
    input_features=input_features,
    output_features=output_features,
)
policy = ACTPolicy(cfg)
 
delta_timestamps = {
    "action": [
        index / metadata.fps
        for index in cfg.action_delta_indices
    ],
}
dataset = LeRobotDataset(dataset_id, delta_timestamps=delta_timestamps)

The full files are training ACT and running ACT on an SO-100. The inference file also shows an operational constraint that is easy to miss: camera names and resolutions must match the model's training metadata.

Diffusion Policy: denoise a future action sequence

Diffusion Policy also models probability(action_chunk | observation_history), but its generator starts from a noisy action chunk and iteratively denoises it. It never wastes compute regenerating the observation.

The training equation in the source is this program:

observation_window, action_chunk = sample(dataset)
t = random_diffusion_step()
noise = standard_normal_like(action_chunk)
noisy_actions = (
    sqrt(alpha_bar[t]) * action_chunk
    + sqrt(1 - alpha_bar[t]) * noise
)
 
predicted_noise = model(
    noisy_actions,
    diffusion_step=t,
    condition=observation_window,
)
loss = mse(predicted_noise, noise)

Conditioning on several previous observations helps infer motion and hidden state. Predicting several actions helps the generator commit to one mode rather than alternate between incompatible moves.

The convolutional architecture embeds noisy actions, images, and poses, then predicts noise at every denoising level. At runtime it starts with random actions and repeatedly subtracts predicted noise. The tutorial notes that a complete chunk can be produced in as few as ten steps.

The Diffusion Policy experiments cover simulated and physical manipulation, including pouring sauce and unrolling a yoga mat. The tutorial reports that useful policies can be trained from 50–150 demonstrations, about 15–60 minutes of teleoperation; that high-frame-rate RGB can approach state-based policy success in tested settings; and that Diffusion Policy beat the considered baselines across the tested dataset sizes. Those are empirical ranges, not a universal sample requirement.

DDIM makes denoising deterministic while targeting the same final distribution and is reported here to use ten times fewer inference steps than DDPM. A Transformer noise model performed strongly but was more sensitive to hyperparameters. The authors recommend starting with the convolutional U-Net; its downside is a bias towards low-frequency functions, which can make non-smooth action sequences harder to fit.

The LeRobot implementation mirrors ACT's training pipeline but uses DiffusionConfig and includes both state-history and action-chunk timestamps. I would read the full Diffusion Policy training snippet and SO-100 inference snippet. Both save preprocessing and postprocessing state with the checkpoint, because normalisation is part of the deployed policy, not an incidental training detail.

Asynchronous inference without idle robots

Separate planning from execution

Action chunks create a deployment choice. The robot can execute all H actions from one observation, which is cheap but open-loop. It can ask the model for a new chunk every control tick and average overlaps, which is responsive but compute-heavy. Or it can predict the next chunk while executing the current one.

I would express the asynchronous design as two services:

RobotClient:
    capture observations
    consume one action per control tick
    request a new chunk before the queue empties
    merge an incoming chunk with actions not yet executed
 
PolicyServer:
    receive observation
    run policy on stronger compute, possibly remotely
    return [action_0, ..., action_H]

The source algorithm is terse, so I would implement its control flow this way:

observation = robot.observe()
queue = await policy_server.predict(observation)
request = None
last_sent_observation = observation
 
while not done:
    robot.execute(queue.pop_left())
 
    remaining_fraction = len(queue) / actions_per_chunk
    if remaining_fraction < threshold_g and no_request_in_flight:
        new_observation = robot.observe()
 
        if (
            joint_distance(
                new_observation.joint_state,
                last_sent_observation.joint_state,
            ) > duplicate_limit
            or len(queue) == 0
        ):
            request = policy_server.predict_async(new_observation)
            last_sent_observation = new_observation
 
    if request is not None and request.is_ready():
        incoming = request.result()
        queue = merge_overlapping_actions(queue, incoming)
        request = None

The duplicate filter matters. If joint state has barely changed, sending another nearly identical observation can produce a nearly identical chunk that repeatedly refills the queue and stalls progress. If the queue empties, the filter is bypassed because no action is worse than a redundant request.

Choose the queue threshold from latency

The source's latency equations become:

round_trip_latency =
    client_to_server_time
    + server_inference_time
    + server_to_client_time
 
expected_round_trip =
    expected(client_to_server)
    + expected(server_inference)
    + expected(server_to_client)

If network time is symmetric and much smaller than model inference, expected_round_trip ≈ expected_server_inference. At 30 fps, one control cycle is 1 / 30 seconds, about 33 ms. To request early enough:

minimum_g = (
    expected_server_inference / control_cycle_seconds
) / actions_per_chunk

For example, a 500 ms inference time consumes about 15 control ticks. With a 50-action chunk, g should be at least 15 / 50 = 0.3 before adding jitter margin.

For deployment, I would calculate that margin from a high latency percentile rather than the mean. If the result exceeds 1, the action chunk is too short to hide the observed latency; no threshold can repair it.

The three limits explain the trade-off:

• With g = 0, the client drains the chunk before requesting another. It then idles for roughly one inference latency.
• With 0 < g < 1, planning overlaps execution and incoming actions are merged across the shared future timesteps.
• With g = 1, an observation is sent at every tick. The source calls this the sync-inference limit, although operationally I find "eager per-tick replanning" less confusing. It maximises reactivity and forward-pass cost.

Without duplicate filtering, requests are triggered every (1 - g) * H * control_cycle seconds and a new chunk arrives after that interval plus average server latency. Filtering stretches the interval when the robot has barely moved.

The actual server is deliberately small:

config = PolicyServerConfig(host=host, port=port)
serve(config)

The client carries the policy and queue details:

client_cfg = RobotClientConfig(
    robot=robot_cfg,
    server_address=server_address,
    policy_type="smolvla",
    pretrained_name_or_path="fracapuano/smolvla_async",
    chunk_size_threshold=0.5,
    actions_per_chunk=50,
)
client = RobotClient(client_cfg)

The full files are policy server and robot client. The client also starts a receiver thread, accepts a language task, runs the control loop, and can plot queue size.

Asynchronous inference hides average latency; it does not remove stale observations, packet loss, jitter, model failure, or unsafe queued actions. I would time-stamp observations and actions, reject chunks older than a limit, bound every action locally, and define a safe stop that does not depend on the policy server being reachable.

Generalist VLAs: pi-zero and SmolVLA

From one policy per task to pretrain and adapt

A specialist policy is trained for one task, robot, and environment. A robot foundation model aims to pretrain across many of them, then adapt to a narrower deployment. That worked earlier in language and vision because Transformers, large datasets, and large compute made reusable representations practical.

Robotics makes the same recipe harder. Data are embodied: locomotion and manipulation have different state and action spaces, and two arms may disagree on joint count, camera placement, control rate, or action meaning. Demonstrations cost more than web text, human operators use different strategies, and naive mixing can cause negative transfer. Cross-embodiment data is therefore both the opportunity and the schema problem.

The progression in the tutorial is useful:

• BC-Z was a latent-variable policy trained on more than 25,000 demonstrations.
• Gato was a broader multi-domain generalist predecessor, although RT-1 used a much larger and more varied robot-task collection.
• RT-1 used a Transformer and roughly 130,000 human demonstrations collected over 17 months with 13 robots and more than 700 language-labelled tasks. The tutorial simplifies its output to six arm values; the paper specifies 11 action dimensions: seven for the arm and gripper, three for the mobile base, and one mode selector, each discretised into 256 bins.
• RT-2 co-fine-tuned VLMs such as PaLI-X and PaLM-E on web and robot data. It represented robot actions as eight-bit tokens in the same output vocabulary as text, turning control into a visual-question-answering-style sequence task. Its reported semantic transfer includes choosing a rock, rather than paper or headphones, as an improvised hammer.
• OpenVLA made a 7B-parameter recipe public. DINOv2 and SigLIP visual features are projected into a Llama 2 7B backbone, which predicts discrete action tokens over 256 levels. It trained on 970,000 Open X-Embodiment episodes.
• π0 and SmolVLA return to continuous action distributions, using action experts and flow matching rather than making every motor value a language token.

Data aggregation was the other half. The tutorial gives 1.4 million trajectories for Open X-Embodiment, while the project and paper use the auditable lower bound of more than one million real-robot trajectories. They confirm 60 source datasets from 34 labs, 22 robot embodiments, and collaboration across 21 institutions. Its RT-X experiments reported positive transfer, where a shared model could outperform some single-embodiment specialists.

DROID standardised distributed collection on a Franka-based stack. The tutorial rounds it to more than 75,000 demonstrations; the project reports 76,000 trajectories, 350 hours, 564 scenes, and 86 tasks. LeRobot adds another route: community members can record and publish compatible episodes on relatively accessible hardware rather than waiting for a central institution.

The modern VLA pattern

A vision-language model usually joins a pretrained vision encoder to a pretrained language model. It first learns from image-text pairs such as LAION-COCO and COYO-700M, plus interleaved corpora such as OBELICS and MMC4, then receives instruction tuning from collections such as LLaVA and Cambrian. The result associates visual patterns with words, object relationships, and broad web semantics. Video and audio can be added through the same broad recipe. Work on SmolVLM, Moondream, MiniCPM-V, and parameter-efficient adapters follows another practical route: shrink the model or update only a small part of it. Robotics still has to ground all of those representations in physical actions, and a robot cannot assume a data-centre budget at inference time.

A modern VLA uses that model as a perception and language backbone, then adds a smaller action-specific module:

images + instruction -> VLM backbone -> semantic features
robot state + noisy action chunk -> action expert -> continuous action chunk

Keeping separate weight sets in a mixture-of-experts-style Transformer avoids spending the full VLM on every motor computation. Here “experts” means two routed parameter streams that meet through attention, rather than a generic sparse router choosing among many interchangeable experts. Action chunking reduces step-by-step drift; a diffusion or flow objective represents multiple valid continuous behaviours.

I see three genuine benefits: the backbone does not relearn every object concept from robot data, the generative expert can model multimodal human demonstrations, and specialised weights can reduce inference cost. I also see three caveats: web semantics do not guarantee contact competence, heterogeneous robot data can conflict, and a large VLM can still be too slow or power-hungry for an onboard loop.

π0: a large VLM plus a continuous action expert

π0 is one concrete architecture. The primary paper describes a unified 3.3B-parameter Transformer with two disjoint experts:

• A PaliGemma 3B VLM, built on Gemma 2B, processes several camera frames and the language instruction.
• A roughly 300M-parameter action expert processes robot state, noisy action tokens, and the flow timestep.

Each expert creates its own query, key, and value matrices. They share information at self-attention layers but retain separate weights.

Its blockwise causal mask is easier to understand as an access-control list:

can_attend_to = {
    "image_and_language": ["image_and_language"],
    "proprioception": ["image_and_language", "proprioception"],
    "actions": ["image_and_language", "proprioception", "actions"],
}

Tokens inside each block attend bidirectionally. Across blocks, later information is hidden from earlier blocks. The VLM therefore does not consume robot-state and action tokens that are far outside its web-pretraining distribution, while the action expert can consume all earlier context. Image, language, and state keys and values can be cached across flow steps.

The tutorial and π0 paper print a path from noise to action but give the vector field with the opposite sign while also showing forward integration. I would not copy that sign mismatch into an implementation. A coherent noise-to-action convention is:

tau = s * (1 - beta_sample(a=1.5, b=1.0))
noise = standard_normal_like(expert_action_chunk)
noisy_chunk = tau * expert_action_chunk + (1 - tau) * noise
 
target_velocity = expert_action_chunk - noise
predicted_velocity = action_expert(
    noisy_chunk,
    observation=current_observation,
    flow_time=tau,
)
loss = mse(predicted_velocity, target_velocity)

Equivalently, one can keep the printed noise - action target and integrate from clean data towards noise, or use a negative solver step. The path derivative, target sign, and solver direction must agree. Both the VLM and action-expert parameters were updated by the objective in the original recipe. Later knowledge-insulation work found that naively sending continuous action-expert gradients into the pretrained VLM can harm training speed and knowledge transfer, and proposed blocking that gradient path while adapting the backbone through discretised actions.

Runtime generation starts with noise and numerically integrates the predicted field:

action_chunk = standard_normal(action_chunk_shape)
for _ in range(10):
    action_chunk = (
        action_chunk
        + step_size * action_expert(action_chunk, observation)
    )

The tutorial reports as few as ten flow steps. Instead of sampling flow time uniformly, π0 uses tau = s * (1 - Beta(1.5, 1)), which emphasises lower, noisier timesteps. The paper sets s = 0.999; omitting values above that cutoff is valid when the integration step is larger than 1 - s, and still allows up to 1,000 steps. The intent is to spend less training on easy near-identity states and more on reconstructing action from informative observations.

The tutorial describes π0's scale as “more than ten million trajectories”, but the primary paper uses different units: about 10,000 hours of robot data and 903 million proprietary timesteps, split into 106 million single-arm and 797 million dual-arm steps. Public datasets including Open X-Embodiment, Bridge v2, and DROID make up 9.1% of the mixture; the private data spans seven robot configurations and 68 broadly defined tasks. Pretraining followed by fine-tuning on narrower, high-quality data consistently beat training the same task from scratch in the reported benchmarks. The authors' explanation is practical: polished task data often omits near-failures and recoveries, while a large, messier corpus can contain those useful states.

Cross-embodiment output uses the largest action dimension in the mixture and zero-pads robots with fewer degrees of freedom. π0 expects exactly three image slots and masks missing views. The real LeRobot π0 inference snippet reflects that contract with base, left-wrist, and right-wrist camera keys, plus task and robot_type fields. A model may be generalist, but its input schema is still exact.

SmolVLA: make the same pattern deployable

SmolVLA keeps the broad pattern but targets accessible hardware and open community data. It uses SmolVLM-2 as the backbone, with SigLIP visual features feeding a SmolLM2 language decoder, and adds an action expert of roughly 100M parameters. The total model is about 450M parameters rather than π0's 3.3B.

The input is multi-view RGB, a natural-language instruction, projected sensorimotor state, and a noisy action chunk. Robot state first enters the VLM token width, then is projected down to the action expert, whose embedding width is 0.75 * VLM_width.

The attention design differs from π0:

• SmolVLA uses a simple causal mask rather than π0's blockwise mask.
• The action expert alternates self-attention and cross-attention. In self-attention, action tokens supply queries, keys, and values. In cross-attention, action tokens supply queries while visual, language, and state features supply keys and values.
• Those context keys and values can be cached across flow steps.
• Pixel shuffling reduces each frame to 64 visual tokens.
• Only the first half of the VLM decoder layers feed the action expert, cutting much of the backbone compute.

The tutorial rounds the pretraining corpus to more than 450 community datasets and more than 20,000 trajectories; the SmolVLA paper reports 481 datasets and roughly 23,000 trajectories. Missing or noisy instructions are re-annotated by a small off-the-shelf VLM using sampled frames, and camera views are normalised into top, wrist, and side roles. This cleanup is essential: a generalist model cannot infer that camera1 means wrist in one dataset and overhead in another without a mapping.

Like π0, SmolVLA uses ten flow-integration steps. The paper reports roughly 40% faster training and six times lower training memory than π0, with competitive results in its tested simulated and real environments. The Hugging Face release separately reports asynchronous inference completing tasks about 30% faster in its test and doubling completions in a fixed-time setting. I treat those as benchmark-specific results, not evidence that a 450M model is universally better than a larger one.

The real SmolVLA inference snippet is short enough to show the runtime boundary:

model = SmolVLAPolicy.from_pretrained("lerobot/smolvla_base")
observation = robot.get_observation()
frame = build_inference_frame(
    observation,
    dataset_features,
    device,
    task=task,
    robot_type=robot_type,
)
action = model.select_action(preprocess(frame))
robot.send_action(make_robot_action(postprocess(action), dataset_features))

I like SmolVLA's direction because the data, model, training recipe, and inference path are inspectable. Openness does not prove generalisation, but it lets a programmer find out where it breaks.

My verdict: I would begin robot learning from the data and runtime loop, not from probability notation. Once observation -> action -> next observation is clear, the maths describes implementation choices: reward accumulation, bootstrapped value targets, reconstruction plus regularisation, noise prediction, or a vector field integrated over time.

I would also keep the original tutorial's restraint. Learning is strongest where hand-written models become brittle: raw perception, varied contact, many tasks, and large demonstration corpora. Classical robotics remains strongest where geometry, constraints, feedback, safety, and known dynamics can be stated directly. The practical system is usually a careful composition of both.

The final lesson for me is openness. Shared formats, inspectable datasets, released weights, complete recipes, and accessible hardware do not solve generalisation by themselves. They make claims reproducible and let many programmers contribute data or discover failure modes that one central lab would miss.