Physical AI Training Data: Why Humanoid Robots Need More Than Video

I watched Gemini Robotics 2 close a zip bag with a 22-degree-of-freedom robot hand. The fingers found both sides of the seal, pinched them together, and ran along the plastic. It looked like a small domestic victory.

Then I watched it again and noticed how little the video explained. The hand hid the contact point. The clip revealed no finger forces, command timing, failed attempts, or corrections. Google says multifinger dexterity remains challenging, which is easy to believe: a zip bag bends, buckles, and slips for reasons a camera may never see.

Physical AI training data is the record that sits behind such an attempt. It connects what the robot sensed, what it was asked to do, how its body moved, and what happened next. Video can show the ending. Training requires the events that made the ending possible.

Suppose a team starts with that clip and a different humanoid. They have another hand, another controller, and a table covered with identical bags. Their job is to produce the second successful attempt. Everything that follows is a data problem.

The team begins with the perfect ending

The clip gives our imaginary team a goal. It shows the bag on a table, the hands approaching from opposite sides, and the seal closing from one end to the other. A model can learn a great deal from that visible sequence.

Scale makes human video even more attractive. Ego4D contains 3,670 hours of egocentric data from 923 participants. It records hand-object interactions, changing object states, and daily activity across far more situations than one robotics team could stage. A model trained on such footage can learn what kitchens look like, how lids turn, where people grip containers, and what usually happens next.

Our team adds that knowledge to its model. The humanoid can now recognise the bag, understand the instruction, and predict that the two sides of the seal must meet. This is real progress. It is also where the clip stops helping.

The person in the source video does not come with a Unitree action log. There are no robot joint states, motor commands, control delays, or measurements from the hand. Our guide to egocentric data for robot learning examines that gap in detail. For this attempt, the consequence is immediate: the model understands the task but has not learned the movement of the body in front of it.

Google reports that Gemini Robotics On-Device 2 can adapt to new bi-arm robots with fewer than 200 examples and a few hours of training. That sounds wonderfully small. It also confirms that some examples from the new body are still needed. Our team has reached the first obstacle: it owns the ending, but not the action that gets there.

The robot inherits the wrong body

A vision-language-action model turns visual observations and language instructions into actions a robot can execute. The phrase makes the hand-off sound tidy. It is not. The action depends on the machine.

RT-2 represented actions as discrete tokens. π0 uses flow matching to generate continuous action chunks. Another policy may predict joint targets, hand commands, end-effector motion, or whole-body actions for a specific controller. The same visible reach can require a different label on every robot.

Our team tries pose tracking and motion retargeting. The human hand in the video becomes a sequence of target poses for the humanoid. On screen, the motion looks plausible. On hardware, one wrist approaches its joint limit and the fingertips meet the bag at the wrong angle. The robot has copied the shape of the movement without inheriting the body that made it work.

Even a recorded command can mislead. Safety clipping, latency, balance control, and contact may change the motion after the request is issued. If the log stores only what the interface requested, it can teach the model an action that never happened.

The team now needs to know what its own robot actually sees and does. That means putting the body inside the collection loop.

Reality makes the lesson expensive

A robot demonstration records an example as aligned observations, states, actions, and outcomes. With teleoperation, a person supplies the strategy while the robot supplies the body. The cameras, joints, controller, and hand all belong to the machine that must eventually work alone.

DROID shows what this costs at useful scale. Its team collected 76,000 demonstrations, or 350 hours of interaction, across 564 scenes and 86 tasks. Fifty collectors used a shared Franka arm and sensor setup over 12 months. The result contains what the internet clip did not: calibrated observations paired with actions a real robot executed.

Twelve months, fifty collectors, one hardware design. The numbers explain why robotics cannot casually repeat the data recipe used for language models. Reality must be present for every physical demonstration, waiting while the robot acts, fails, resets, and tries again.

Broader datasets can spread that cost. Open X-Embodiment pooled more than one million trajectories from 60 datasets and 34 laboratories across 22 robot embodiments. Its RT-X results showed that cross-embodiment data can improve a target policy, especially when target data is scarce.

Our team gladly borrows those priors. It still cannot borrow another hand’s joint limits, camera position, or contact geometry. The zip bag must eventually be closed by the hand attached to the humanoid in the room. Real data provides that grounding, but collecting a new physical attempt for every variation would take far too long.

The next temptation is obvious: manufacture the attempts.

The team manufactures 780,000 more attempts

Synthetic data lets the team move the bag, change the lighting, alter the approach angle, and repeat the task without asking an operator to reset a physical table. A simulator can also reveal exact object poses, contact states, and labels that are difficult to measure in reality.

NVIDIA reports that its synthetic motion pipeline generated 780,000 trajectories in 11 hours. Mixing those trajectories with real data improved GR00T N1 performance by 40% over real data alone in the company’s experiment. Our analysis of NVIDIA Isaac GR00T looks at that stack more closely.

Our imaginary team follows the same principle. It seeds simulation with real motion, varies the bag position and scene, then trains across thousands of synthetic closures. The policy becomes less brittle. It reaches from new angles and copes when the bag starts a few centimetres away.

Then the physical robot tries again. The fingers meet the plastic, but the seal slips. The simulator generated exact labels for the contact model it was given. It could not announce that its friction, compliance, or sensor noise was wrong.

This is the blunt limit of scale. Another million simulated episodes can create another million confident examples of the same mismatch. Sim-to-real transfer has to be measured on representative hardware. Simulation has carried the policy back to the table, but the next lesson is hiding underneath the fingertips.

The first failure finally tells us something

The failed closure is more useful than it looks. The camera shows the right pose, yet the bag escapes. That contradiction tells the team to inspect a signal the video never contained: touch.

A 2026 multi-resolution tactile imitation-learning study tested five contact-rich tasks on a Franka arm using 30 expert demonstrations. Its multi-tactile policy averaged 80% success, compared with 31% for the vision-only baseline and 54% for a visual-tactile comparison. The study is a narrow preprint, not a universal rule for humanoids. It does show that pixels can become least informative at the moment contact decides the task.

The team adds tactile readings and keeps the failed attempts. It records the commanded motion, executed motion, finger state, intervention, and reset. Now the data explains when the seal begins to slip and what correction follows. The failure has become supervision.

This is where a demo reel and a training dataset part company. The editor cuts when the bag closes. The dataset should preserve the miss before it, the recovery after it, and the safety stop when recovery fails. Synchronisation, calibration, provenance, and embodiment metadata keep those events in the right order. Without them, nobody can tell which action caused which result.

On the next attempt, perhaps the robot closes the bag. Perhaps it fails differently. Either outcome extends the record because the team can connect perception, action, contact, and consequence on the real machine. Our guide to evaluating humanoid robot training data covers the wider diligence questions, but this causal chain is the part I would refuse to lose.

The second attempt

Back at the table, the robot reaches for another bag. It may close the seal. It may fail in a new way. The difference is that the team can now explain the attempt: what the hand saw, which action was requested, what the controller executed, where contact changed, and why the robot continued or stopped.

That is the standard I would use for physical AI training data. There is no universal ratio of human video, robot demonstrations, simulation, and tactile data. The right mixture is the one that keeps the chain from instruction to physical consequence intact for the body performing the task.

The Humanoids Data dataset catalogue separates real robot data, human demonstrations, simulation, teleoperation, and tactile or force collections so the full attempt remains visible.

The original clip showed how the story ended once. The full record gives the robot a credible way to begin again.