The Robotics Data Pyramid: Web, Synthetic, and Real Robot Data

The robotics data pyramid groups robot training data into three broad tiers. Web and human data form the wide base, synthetic robot data sits in the middle, and real-robot data occupies the apex. Typical data volume decreases as you move up, while robot alignment and embodiment specificity increase.

This three-tier version follows the data pyramid used to train NVIDIA's GR00T N1. It is useful beyond one model because it exposes the central trade-off in physical AI: plentiful data usually provides indirect supervision, while directly executable robot experience remains expensive to collect.

The pyramid is a taxonomy of data sources, not a complete training or deployment lifecycle. Cross-robot trajectories, target-robot demonstrations, autonomous rollouts, interventions, and recoveries are all forms of real-robot data. Evaluation belongs outside the training pyramid so that it can measure whether a new data mixture actually improved the system.

In brief

• Web and human data provide broad visual, semantic, and behavioural priors, but usually lack executable robot actions.
• Synthetic data adds controllable variation, robot-shaped trajectories, and labels that are difficult to capture in reality, but inherits simulation or generation errors.
• Real-robot data records physical execution. It includes shared cross-embodiment datasets, target-robot demonstrations, autonomous experience, failures, and corrections.
• UMI-style demonstrations and egocentric video sit between the broad categories because they preserve more action information than ordinary video without operating the target robot.
• There is no universal layer ratio. Every source needs clear timing, semantics, provenance, and rights, and the final mixture needs held-out evaluation.

How to read the robotics data pyramid

The width of each tier represents typical availability and collection scalability, not a required number of samples. A deployed fleet can produce more real logs than a small research team has web video. The vertical direction represents how directly the data describes executable behaviour on a physical robot.

This is not a quality ranking. Irrelevant real-robot repetition can be less useful than diverse simulation, and physically implausible synthetic data can be worse than a small clean demonstration set. A July 2026 preprint, Data Pyramid for Embodied Manipulation, expands the idea into five complementary sources—general data, simulation, egocentric and exocentric video, UMI-style demonstrations, and real-robot data—and evaluates them across scalability, robot alignment, quality, diversity, reusability, and physical fidelity. The ordering is a trade-off, not a strict progression on every property.

Base: web and human data

The base includes general image, language, and video corpora as well as first-person and third-person recordings of people interacting with the world. These sources can teach object identity, spatial relationships, task language, hand-object interaction, motion, and likely future states across far more situations than a robot fleet can practically collect.

What they usually do not contain is the action that a particular robot should execute. Human video has no robot proprioception, motor command, controller response, or calibrated contact signal. A model may see a person close a drawer without knowing the joint targets, force, latency, or collision constraints needed to reproduce the result.

Models use this tier in different ways. Some learn representations or world dynamics from action-free video. Others infer latent or pseudo-actions so video can enter a robot-training objective. GR00T N1, for example, uses a learned latent-action codebook and an inverse-dynamics model to annotate action-less human and generated video. Those inferred actions are useful supervision, but they are not equivalent to measured commands from a physical robot.

Meta's V-JEPA 2 study illustrates the distinction. The researchers pretrained on more than one million hours of internet video, then used fewer than 62 hours of action-conditioned DROID robot video for planning. The result applies to that model and arm setup, not to a universal data ratio. It shows that abundant observation can provide broad physical priors while robot interaction supplies the missing action grounding.

Middle: synthetic robot data

The middle tier contains data produced computationally for robot learning. It includes trajectories generated in physics simulation and neural trajectories generated or augmented by video models. Unlike ordinary web video, synthetic pipelines can expose actions, object poses, contacts, segmentation, rewards, and success conditions at every step.

Simulation is valuable because tasks can be repeated in parallel while changing object pose, lighting, camera position, friction, mass, or failure conditions. Neural generation can expand visual and behavioural variation around real initial frames and language instructions. A generated video becomes a robot trajectory only when the pipeline can associate it with a usable action representation, often through simulation state, a latent-action model, or an inverse-dynamics model.

The scale advantage is real, but so is the fidelity limit. A simulator approximates contact, deformable objects, sensor noise, control delay, and hardware wear. A video model can generate motion that looks plausible while violating geometry or dynamics. Synthetic data therefore expands coverage; it does not independently prove that a policy works on hardware.

An NVIDIA technical blog reports that its GR00T-Mimic synthetic motion pipeline generated 780,000 trajectories in 11 hours and produced a 40% performance improvement when synthetic and real data were combined. The blog does not define that headline as a universal cross-task measure. In a separate low-data neural-trajectory ablation, the GR00T N1 report reports a 5.8-point average gain across eight GR-1 tasks. Both are NVIDIA-reported results for specific workflows, not a general conversion rate between synthetic and real episodes. Their useful lesson is that synthetic data can multiply a small real seed set when the task, action labels, and transfer process are controlled.

Apex: real-robot data

Real-robot data records observations, states, actions, and consequences from physical hardware. It can come from teleoperation, scripted controllers, autonomous rollouts, human interventions, or deployment. This tier supplies the strongest evidence about what a robot actually saw, which command it received, how its controller and body responded, and what happened in the real environment.

The apex still contains a wide range of alignment. Open X-Embodiment pooled more than one million trajectories from 60 datasets and 22 embodiments; its RT-X experiments found positive transfer on multiple evaluated robots, though not uniformly across every domain. DROID used a more consistent Franka setup to collect 76,000 demonstrations, or 350 hours, across 564 scenes and 86 tasks. The Open X-Embodiment and DROID catalogue entries describe their modalities and source terms.

Both are real-robot data, but neither is automatically equivalent to data from the final machine. Camera placement, gripper geometry, joint limits, action spaces, control frequency, and latency all affect transfer. Target-robot demonstrations remain the most directly aligned subset because the eventual sensors, kinematics, controller, and contact geometry are inside the collection loop.

For humanoids, embodiment metadata must extend beyond the hands. Head, torso, base, and foot state can change both the camera view and whether a reaching motion is balanced. A manipulation clip that omits whole-body state may hide the mechanism that made the action feasible.

Data sources that sit between tiers

The three-tier pyramid is deliberately coarse. Some important sources combine properties from adjacent levels:

• Egocentric data records real human-object interaction from a useful viewpoint, but normally lacks robot-ready actions.
• Universal Manipulation Interface demonstrations use portable camera-equipped grippers to capture wrist views, end-effector motion, and gripper width without operating a robot at the collection site. They preserve more action structure than ordinary human video but still require calibration, retargeting, and embodiment-specific deployment. The catalogue includes the UMI dataset.
• Cross-embodiment data is real-robot data, but its action labels may be less aligned with the target body than a smaller target-robot dataset.
• Neural trajectories can begin from real robot frames and imitate the target embodiment while remaining synthetic because the resulting motion was generated rather than executed.

These boundary cases are why the pyramid should guide questions, not assign absolute value. For each source, ask what was observed, whether actions were measured or inferred, which body produced them, and how physical validity was checked.

Build a data mixture, not a layer contest

A useful training recipe assigns a job to each source:

1. Define the target tasks, operating conditions, failure costs, and held-out evaluation before choosing a mixture.
2. Use web and human data for missing semantic, visual, and behavioural breadth.
3. Use simulation or generated trajectories for controlled variation, rare conditions, and scalable robot-shaped supervision.
4. Use compatible cross-robot datasets for reusable action experience, then collect target-robot data for the remaining embodiment gap.
5. Add autonomous failures, human corrections, and recoveries from states the current policy actually visits.
6. Measure the new policy on evaluation data that training has not absorbed.

The mixture cannot be assembled by file concatenation alone. Physical Intelligence reports that its π0.7 model combines data from different robots and control modes, human video, and autonomous episodes, but that naive merging performs poorly. Its prompting framework adds task context, visual subgoals, control-mode labels, speed, and quality metadata to disambiguate different behaviours. That is a source-reported design finding, but the underlying problem is general: two trajectories can look similar while representing different goals, controllers, or levels of proficiency.

There is no universal ratio among the tiers. The right mixture depends on the model objective, action representation, target embodiment, task distribution, and unresolved failure modes. A warehouse arm, a household mobile manipulator, and a humanoid should not be expected to use the same recipe.

Requirements that cut across every tier

Calibration, synchronisation, schemas, provenance, and usage rights are not a separate foundation layer. They determine whether data in every tier can be interpreted and combined. A training-ready record should make clear:

1. The task or instruction and how episode boundaries were chosen.
2. The available observations, state, actions, rewards, and outcomes—and which signals are absent or inferred.
3. The timestamps, coordinate frames, units, calibration, and control frequency.
4. The robot or human embodiment, sensors, controller, software version, and environment.
5. Whether an action is an operator input, requested command, clipped command, or measured execution.
6. Success, failure, intervention, recovery, and reset labels where they exist.
7. The source, consent, licence, and permitted uses.

Sensor calibration and data synchronisation are causal requirements. A camera frame paired with a stale action can teach the wrong relationship. A Cartesian delta without its coordinate frame can mean a different motion on another platform.

Common formats help but do not solve semantics. LeRobotDataset v3.0 can package state and action data in Parquet, video in MP4, and episode metadata in structured files. Two columns named action can still mean joint targets, end-effector deltas, operator inputs, or reconstructed motion. The schema makes a difference visible only when the producer documents it.

Deployment closes the loop outside the pyramid

Autonomous rollouts, failures, and human interventions are not a fourth tier above real data. They are especially informative real-robot data because they sample states created by the policy's own mistakes. Once selected for training, they belong at the apex.

Learning While Deploying, a 2026 preprint, demonstrates this as a data flywheel: a fleet of 16 dual-arm robots contributed autonomous rollouts and optional human interventions to a shared replay buffer, the policy was updated, and new checkpoints were redeployed. Its authors report an overall average score of 0.95 across eight tasks. That average combines binary success on four grocery-restocking tasks with step-wise progress scores on four long-horizon tasks, so it should not be read as a 95% success rate across all eight. This is evidence for one system, not a guarantee that unfiltered deployment logs improve every policy.

Held-out evaluation must remain separate from that flywheel. If failed test episodes enter training while the same scenarios remain in the benchmark, the score can rise because the test was memorised. A sound pipeline keeps:

• A frozen or carefully versioned evaluation set.
• A replay pool of deployment episodes eligible for training.
• Traceable links from each training slice to the failure mode it is intended to address.

A practical checklist for evaluating a dataset

When buying, licensing, or selecting robotics data, ask:

• Which pyramid tier does the source occupy, and what role should it play in training?
• Are actions executed, commanded, reconstructed, latent, or absent?
• How close are the embodiment, sensors, controller, and environment to the target system?
• Which tasks, objects, scenes, outcomes, failures, and recoveries are covered?
• How were timing, calibration, coordinate frames, and episode quality validated?
• Can the data legally be used for the intended training, redistribution, and commercial deployment?
• What held-out evidence will show that adding it improved the robot?

The guide to evaluating humanoid robot training data expands these questions into a diligence process. The article on physical AI training data explains in more detail why video alone cannot preserve the full chain from instruction to physical consequence.

The pyramid's durable lesson is not that one tier wins. Web and human data supply breadth, synthetic data supplies scalable controlled experience, and real-robot data supplies physical grounding. Progress comes from preserving the differences among them, aligning them deliberately, and proving on untouched evidence that the resulting robot improved.