Can Human Motion Become Humanoid Robot Training Data?

Human movement can teach a humanoid robot, but a video is not automatically a robot demonstration. A robot needs a time-aligned motion that fits its body, contacts, and surroundings, plus evidence that the resulting motion can be executed. Two September 2026 studies show different ways to bridge that gap: one predicts robot motion directly from monocular video, while another captures human demonstrations together with the geometry of a cluttered scene.

Both approaches make human movement more reusable. Neither suggests that collecting more video, on its own, removes the work of grounding that movement in a particular robot and task.

Human motion and robot motion are different data

A common pipeline first estimates a person’s body pose from video, then uses motion retargeting to map that pose to a robot. The conversion has to account for different limb lengths, joint layouts, balance, and actuator limits. A human pose that looks plausible can produce a robot trajectory that is unstable or impossible to follow.

The BeyondRetarget preprint, submitted on 24 September 2026, targets an error in that two-stage approach: errors in the estimated human motion can carry into the retargeting step, which cannot go back and use the original pixels to correct them. Its model maps monocular RGB video to robot-specific motion without an explicit human-pose representation at inference.

“Direct from video” describes that inference path. It does not mean the training system learned from video alone.

Video input still depends on motion and contact labels

BeyondRetarget’s experimental setup uses MotionPRO, which the authors describe as synchronized RGB video, optical motion capture with SMPL ground truth, and foot-contact labels. The MotionPRO dataset paper describes 70 volunteers performing 400 motion types across 12.4 million pose frames. The BeyondRetarget paper reports that foot-contact labels supervise its contact predictor.

This pairing is the important distinction. The robot can receive video as its input at test time, while the model learns from richer signals that show how a person moved and when their feet supported them. Those signals help train a robot-oriented representation and a contact-aware refinement stage. A team collecting only RGB clips would need a different way to obtain or learn the motion and contact supervision.

The model predicts a root path, orientation, and joint targets for one robot. A contact-aware step refines the feet; a separate motion tracker then follows the reference. This is motion generation and tracking, not a policy that selects a task from language or object goals. Dataset delivery should label observations, reference motions, commands, and policy rollouts separately; they are different learning targets.

The authors report tests across eight humanoid morphologies. Their method uses robot-specific decoder branches, and adapting a new robot involves retraining those branches. That is evidence of a reusable approach across the robots they tested, not zero-shot support for any humanoid.

The result should also be read at the right level. On the Unitree G1 benchmark, the paper reports that 347 of 362 motion clips completed in MuJoCo simulation, or 95.86%. That is a simulated clip-execution rate, not a real-world task-success rate. The paper also includes hardware comparisons on selected motions, but does not report the same 362-clip success measure on physical robots. These are author-reported results in a new preprint.

The project page describes its live demo as a base version that does not yet support moving-camera scenarios or large-scale global trajectories. The paper also lists complex terrain and contact-rich human-environment interaction as limitations. A short, fixed-camera movement and a long action through a changing scene are different data problems.

Scene geometry changes what a demonstration teaches

Moving Through Clutter (MTC), a 17 September 2026 preprint, takes a different route. Its VR capture system records human movement in procedurally generated cluttered environments. The resulting dataset contains 1,185 demonstrations across 327 environments. The authors then retarget each demonstration into a humanoid reference trajectory while enforcing clearance from the robot’s surroundings, and use those trajectories to train a scene-aware locomotion policy for a Unitree G1.

For tasks such as ducking, crawling, or stepping through a narrow gap, the scene is part of the demonstration. A body trajectory without obstacle geometry cannot show how much clearance was available or whether the robot’s body can fit through the same path. MTC’s dataset and policy are therefore tied together by the recorded environment, not just by the performer’s movement.

That scene link changes retargeting. MTC-SAMR optimizes trajectories over a look-ahead window and enforces clearance constraints, so neighboring frames are solved together rather than as isolated poses. This helps catch a path that looks feasible in one frame but clips an obstacle as the robot moves.

The authors also state a boundary: their current VR capture lacks force feedback, so the pipeline focuses on non-contact traversal. It does not establish that the same demonstrations teach a robot to push, pull, or manipulate an obstacle. The distinction resembles other robot data pipelines: observations, actions, contacts, and outcomes are separate signals, as described in our guide to robot training data modalities.

What makes the data useful for a robot

When evaluating human-motion data, trace one episode from capture to robot test. The questions that matter are practical:

One reviewable episode should keep an ID across the video, source motion, robot target, scene, and evaluation record, and label each field as measured, estimated, or generated. Pressure-derived contact labels are not force readings from the target robot. Our camera and action timing guide explains why timestamps belong in that record.

• What was actually recorded? Check the video viewpoint, frame timing, camera movement, calibration, scene geometry, and whether the human motion was measured or estimated. First-person capture has different strengths and limits from external video or immersive VR; our egocentric data guide explains that distinction.
• What is paired with the video? Look for a time-aligned source motion, robot target trajectory, contact labels, or other supervision. If a pipeline derives targets later, ask which robot pose or action representation it produces and how that conversion can be inspected.
• Which robot does the motion describe? Record the model, joint layout, control interface, and any retargeting or decoder version. A trajectory for one body is not automatically valid for another.
• What was tested? Separate pose-estimation scores, simulated tracking, real-robot motion, and task completion. Report the trial count and failure rule for each. A camera-to-motion demonstration or a successful simulator rollout is useful evidence, but it does not establish performance in a new environment.

These checks extend the questions in our guide to evaluating humanoid robot training data. They also make it easier to compare video-based supervision with teleoperation data, where the robot’s commands and measured state may be captured directly.

More human motion needs more grounding

The current research points toward a practical distinction: human recordings can expand the motions a robot can learn from, but their value depends on the links between the source recording, the target robot, and the evaluation. BeyondRetarget pairs video with motion-capture and contact supervision; MTC pairs immersive demonstrations with scene geometry and robot-clearance constraints.

For a dataset buyer or builder, the useful sample is not a video in isolation. It is one traceable episode showing the source, the motion or action derived from it, the robot model and constraints used, and the conditions under which execution was checked. If those pieces are missing, the footage may still help with visual pretraining or task recognition, but it should not be described as a validated humanoid demonstration.