What 30,000 Hours of Egocentric Video Still Miss

What does a world model learn when its human-video data grows from 300 to 30,000 hours? It gets much better at putting hands where they should be. It remains far worse at predicting what those hands do to the objects they touch.

That is the central result of What 30,000 Hours of Ego-centric Video Does Not Teach, submitted on 8 October 2026. The authors measure interaction fidelity—agreement between a predicted interaction and the recorded physical outcome—rather than treating a plausible-looking video or downstream task score as sufficient. Their fitted ceiling for the agent is 0.800–0.811 Structural Consistency Score (SCS), depending on conditioning. The object ceiling is 0.565. More video improves both, but it does not make them converge.

A plausible hand is not a correct interaction

Most video metrics mix several errors together. A prediction can have believable lighting and texture while the wrong hand moves, or it can place the hand correctly while a spoon stays on the plate instead of rising with the grasp. A robot-facing evaluation needs to separate the actor from the actor's effect on the world.

The paper uses SCS to do that. It segments and tracks hands and manipulated objects through predicted and ground-truth clips, then averages mask intersection-over-union. Agent SCS measures the hands; object SCS measures the things they manipulate. LPIPS is reported separately as a perceptual control, so a model does not receive credit for merely producing a prettier wrong interaction.

SCS is not a direct physics measurement. Mask tracking is imperfect, and the authors estimate that even ground-truth videos top out near 0.93 for hands and 0.90 for objects under the measurement pipeline. Nor does one overlap number explain whether an error came from missed contact, incorrect stiffness, a disappearing object, or a bad hand trajectory. Its value here is diagnostic separation: it reveals that agent and object predictions follow different scaling curves.

This question is different from Figure's Helix 2.5 scaling study. Figure varied human pretraining data and measured held-out robot-action prediction loss before a separate household evaluation. This study predicts video and directly measures the hand and object structures inside it. One asks whether more data improves action prediction; the other asks which visible consequences a generative model still gets wrong.

The 100-fold ladder is richer than plain video hours

The training pool contains 1.15 million first-person clips totalling 30,000 hours, or 3.24 billion frames at 30 frames per second. The paper's dataset description reports 116 environment categories, 1,106 scene categories, 32,051 tasks, and 14,019 participants, with no environment type contributing more than 8.3% of the duration.

Calling this “video” hides important supervision. Each frame also has calibrated camera intrinsics, a 66-degree-of-freedom camera pose, dense 3D hand keypoints, wrist poses, upper-body keypoints, and a free-form activity description. The models are trained at nested checkpoints of 300, 1,000, 3,000, 10,000, and 30,000 hours. Larger checkpoints extend the same ordered data prefix and receive more optimisation steps.

That last detail sets the boundary of the scaling law. Data volume and training compute increase together, so the curve describes joint scaling rather than the isolated value of another hour of footage. The authors note that training smaller budgets to convergence would likely lift the left side of the curve and lower the fitted ceiling, making their ceiling estimates optimistic rather than conservative.

The evaluation is out of distribution with respect to capture hardware, sites, and participants. That is stronger than testing held-out clips from the same recording sessions, but both training and evaluation still come from one collection protocol. The benchmark, masks, code, and checkpoints are promised for release; the project page currently exposes the paper and qualitative media rather than a public download for those artefacts.

A skeleton makes the agent cheaper to model

The decisive control is skeleton conditioning. Alongside the initial frame and text action, the model receives a projected sequence of the person's body configuration in the camera view. This removes much of the need to infer where the actor should move from an action label alone.

For agent fidelity, the ordinary Nano model gains 0.12 SCS from 300 to 3,000 hours, then only 0.06 from 3,000 to 30,000. Its fitted asymptote is 0.811. With skeleton conditioning, the 300-hour model reaches the agent fidelity that ordinary conditioning needs roughly 15,000 hours to attain—a reported 50-fold data reduction—and the fitted limit remains almost the same at 0.800.

Conditioning therefore changes how quickly the model reaches the hand-motion ceiling, not where that ceiling lies. This matters for data procurement. If a more direct pose signal can replace most of the agent-side gain, buying another order of magnitude of undifferentiated video is an expensive way to teach a quantity that could have been supplied explicitly.

Object fidelity heads towards a lower ceiling

Objects also benefit from knowing where the body moves, but much less. With skeleton conditioning, the Nano model's object SCS rises by 0.056 between 300 and 3,000 hours and only 0.035 between 3,000 and 30,000. The fitted asymptote is 0.565, and the 30,000-hour checkpoint has already reached 93% of it.

The paper estimates the metric's achievable maximum separately for each category. Relative to that bound, agent fidelity converges to 86% of its measurable range and object fidelity to 63%. This is not simply a case of objects needing another modest batch. The curves suggest the model is heading towards different limits.

Increasing model size does not offer an economical escape. A fourfold larger Super variant raises the fitted object ceiling by 0.034 SCS while barely changing agent fidelity. At that observed increment, closing the remaining 0.205-point gap would require roughly six further quadruplings—about four orders of magnitude more parameters. The paper calls this a lower-bound estimate because it assumes that each quadrupling keeps delivering the same gain.

Those are fitted author-reported trends, not guarantees beyond 30,000 hours. They are still more informative than saying “objects improve more slowly”, because they put a testable ceiling on the current architecture and supervision.

Changing supervision moves the ceiling a little

The authors next shift training attention towards the changing object. Their object-centric adaptive noise schedule identifies dynamic regions from frame differences and injects more diffusion noise there; a companion weighting term emphasises the same regions in the loss. This makes reconstructing the manipulated object harder to avoid than reconstructing the largely static room.

At 30,000 hours, object SCS rises from 0.527 for the skeleton-conditioned baseline to 0.546, a reported 3.7% improvement. The gain appears across the data ladder and raises the fitted ceiling, but by a roughly constant amount. Reweighting alone does not produce the improvement; the ablation attributes it to the region-specific noise increase, with a small further benefit from combining both components.

The result is useful precisely because it is limited. A training objective can recover more object dynamics from the same footage, so data quantity is not the only lever. Yet 0.546 remains far below the agent ceiling. Better supervision moves the boundary without removing it.

Pixels lose the physics at contact

Two missing signals explain why. First, the next object state depends on mass, friction, stiffness, internal structure, and contact force. A single first-person image does not uniquely reveal those properties. Several physically different objects can look the same before they are touched.

Second, the most consequential geometry is often hidden at the exact moment it matters. A hand creates occlusion over the grasp point; video shows that a hand closed, but not the contact patch, normal force, slip, or pressure distribution beneath it. The paper's project page groups the remaining failures into missed contact, wrong physical properties, lost object permanence, and non-rigid dynamics.

A learned video model and a physics simulator have complementary weaknesses. Video covers diverse real appearances and messy human behaviour, while a simulator exposes geometry, contacts, forces, and material parameters directly. The study does not show that simulation can reproduce every deformable real-world interaction; it shows why pixels alone cannot supply every state variable that a physical forecast needs.

Humanoid transfer preserves the asymmetry

The authors fine-tune three Cosmos 3 Nano variants on 11 simulated bimanual humanoid tasks. All predict 16 future frames from a starting frame and a 138-dimensional humanoid state trajectory, and all are evaluated on policy rollouts rather than demonstrations.

Human pretraining contributes 0.07 hand SCS and 0.03 object SCS. Conditioning and the new supervision add another 0.17 and 0.06 respectively. Success agreement—whether the generated clip supports the same success or failure judgement as the simulator—follows the same ordering.

The absolute object scores are higher than in the human-video benchmark because these simulated objects are relatively large and rigid. The authors explicitly caution that evaluation difficulty changes the conclusion. This is simulation evidence about video prediction around humanoid policy rollouts, not a real-hardware task-success result.

A useful video order needs the missing record

For a data buyer, the paper turns “more diverse video” into a more precise specification. Broad egocentric data can teach scene variety, task sequence, hand approaches, and visible state change. It should not be priced as though every hour contains measured contact physics.

A stronger collection package would preserve:

• unique source duration and clip lineage, so overlapping windows are not counted as independent experience;
• camera calibration, timestamps, poses, and confidence for every derived hand or body track;
• persistent object identities, geometry, pose, material class, and uncertainty through occlusion;
• robot observations, commands, measured state, and outcome labels when human video is grounded to an embodiment;
• tactile or force-torque signals for tasks where contact quality determines the result;
• failures and recoveries, not only clips whose visible outcome matches the instruction.

The distinction also sharpens the site's guide to first-person robot data. Plain footage, tracked human capture, and robot-mounted logs are not interchangeable hours. This study starts from unusually rich tracked human capture and still finds a contact-effect gap. A less instrumented collection should expect that gap to be larger, not smaller.

The result sets a boundary, not a universal law

The experiment covers one generative architecture family, one collection protocol, and one relatively small out-of-distribution evaluation set. Its progressive ladder changes data exposure and optimisation together. SCS depends on segmentation and tracking, and fitted asymptotes depend on the curve continuing beyond the observed 30,000-hour range. Latent-predictive models that do not render pixels may behave differently and cannot be judged with the same metric.

The release boundary matters too. The paper says the benchmark, annotations, code, and checkpoints will be published, but they are not yet linked as downloadable artefacts on the project page. Until they are available, the quantitative results are source-reported rather than independently reproduced.

Within those limits, the study answers its reader question clearly. A 100-fold increase in egocentric video makes the actor much easier to predict and improves the object, but the model approaches a substantially lower ceiling for the object's response. The next useful dataset is not merely a larger pile of pixels. It is a record that makes contact, hidden geometry, physical properties, and executed outcomes less ambiguous.