Can a Humanoid Learn Contact-Rich Skills From Video?

A humanoid can learn a contact-rich skill from an ordinary third-person video, but copying the visible body motion is not enough. The useful part of the demonstration has to be made explicit: which robot link should touch what, at which world-frame pose and time, how quickly it should approach, and which parts of the scene it must avoid.

The VICAR preprint, submitted on 9 October 2026, demonstrates that pipeline on ten skills. It reconstructs selected scene geometry, manually marks task-defining contacts, adapts the motion to a Unitree G1, generates variants around movable contact points, and trains a tracking policy in simulation. The authors report real-hardware trials for table-tennis serves and three pickup tasks. They also report a ladder-climbing hardware failure, substantial motion-capture requirements at deployment, and no force or contact-mode representation in the contact labels.

That combination makes VICAR more useful than a highlight reel. It shows both how one source video can seed a family of robot references and why “learned from video” does not mean video was the only data, supervision, or infrastructure involved.

A similar pose can still miss the task

The embodiment mismatch is easiest to see at a table. A surface near a person’s waist can sit near a shorter humanoid’s chest. If motion retargeting only tries to preserve relative body shape or joint motion, the robot hand can follow a recognisable reach while passing above the object. A collision-free result is not necessarily task-correct either: many paths avoid the table, but only some reach the object from the demonstrated side with a usable approach speed.

Contact-rich loco-manipulation makes that ambiguity worse. In VICAR’s under-table pickup, one hand supports the body on the table edge while the other reaches below it. The motion has to preserve the support contact, the pickup contact, collision clearance, whole-body continuity, and balance-relevant timing. Treating walking, reaching, and touching as independent clips would remove the relationships that define the skill.

VICAR represents a selected contact as a robot body, a frame number, and a desired position and orientation in the task’s world coordinate frame. That is a six-degree-of-freedom pose objective. The paper is explicit about what it is not: the annotation does not encode force, sliding, contact mode, or the physical state of the surface. A labelled racket–ball hit says where and how the racket should arrive, not how much impulse the ball receives.

The video starts a data pipeline rather than ending one

The method section separates the transformation into stages. That separation is important for interpreting what the final policy learned.

1. Recover human motion. GENMO/GEM estimates global body motion, root movement, and task-relevant keypoints from the source video. The current prototype still requires a person to label contact frames, contacted body parts, and contact orientations. The authors report roughly 20–30 minutes of manual annotation per skill.
2. Reconstruct the constraining scene. UniDepth estimates metric depth and Semantic-SAM segments the frame. GPT-4o mini selects regions likely to constrain the task, such as a tabletop, edge, floor patch, or support surface. Those pixels are unprojected, scaled using the recovered person and scene, and voxelised into collision geometry. This is a task-focused partial reconstruction, not a complete digital twin.
3. Retarget in two passes. An OmniRetarget-style first pass supplies a robot motion prior under joint limits. A second optimisation gives high priority to collision clearance and the manually selected global contact poses while also preserving end-effector speed and trajectory smoothness. The paper describes these as large Lagrangian penalties, not hard guarantees; conflicting geometry can leave a residual contact error.
4. Generate contact variants. A motion generator receives a source-motion style and movable contact conditions. For tennis, the hit point moves around a three-dimensional grid. For pickup, the object pose changes within a task-specific range. Each variant goes through the robot-aware optimisation rather than simply translating the original joints.
5. Train and track in physics. A reinforcement-learning policy learns to follow the generated reference in Isaac Lab. It observes the reference, robot state, velocities, gravity, and previous action, while training includes changes to friction, joint offsets, centre of mass, and external pushes.

The result is therefore not one video mapped directly to motor commands. It is a lineage of estimated motion, human contact labels, reconstructed geometry, optimised robot references, generated contact variants, simulator rollouts, and a deployable tracking policy. The original clip is evidence about task structure; the simulator supplies much of the execution experience.

Movable contacts turn one clip into a bounded motion family

The most useful data idea in VICAR is not that one video replaces a dataset. It is that one carefully processed video can define a bounded family of task variations.

For each of six table-tennis serves, the style identifier remains fixed while toss and hit locations vary. Pickup motions are regenerated around different object poses. The bimanual example keeps both hands synchronised with a box, while the ladder example preserves an ordered sequence of hand and foot supports. This is controlled augmentation in task space: the varying quantity has a physical meaning that can be inspected.

The distinction matters for dataset counts. The paper defines a skill as one source video plus its contact-conditioned motion family. Six tennis styles therefore count as six skills even though each can produce many optimised trajectories and simulator episodes. A data catalogue should keep at least three numbers separate:

• unique source videos and their rights or provenance;
• derived robot references, including the sampled contact conditions and optimiser residuals;
• executed simulation or hardware trials, with outcome and measured state.

Calling every generated variant a new demonstration would hide their shared source. Calling the package “ten videos” would hide the physics training and contact augmentation that make it useful.

The simulation result and the hardware result answer different questions

The authors compare retargeting methods while giving the baselines the same contact-pose penalties, augmentation, policy compute, and reinforcement-learning settings. That design aims to isolate the quality of the retargeted reference rather than reward VICAR for extra downstream training.

In 1,000-trial simulation evaluations, the paper reports 99.8% downstream success for VICAR’s tabletop pickup, compared with 76.2% for OmniRetarget. For ladder climbing, the reported comparison is 95.5% versus 56.0%. The claimed advantage is not merely lower penetration: preserving speed and finding a smooth path around the obstacle avoids discontinuities that a later collision correction can create.

Those are source-reported simulator results, not physical-robot success rates. The hardware section reports separate trial counts:

No real-robot data was used to train those policies, so the paper calls deployment zero-shot sim-to-real in that specific training sense. It is not zero-shot perception. An external motion-capture system supplies global ball and object positions at runtime, while the policy tracks references from proprioception and projected gravity. Replacing that laboratory measurement stack with onboard perception remains future work.

The ladder is the most informative failure

VICAR’s ladder reference reached 95.5% reported success in simulation, yet the authors say hardware deployment failed because the blind tracking policy stumbled. That gap prevents an easy claim that smoother retargeting solves contact-rich transfer.

The reference can encode where hands and feet should land, while the deployed system still lacks the perception or feedback needed to recover when a physical foothold, body state, or timing differs from simulation. The authors also report substantial motion-capture engineering to recover global pelvis, object, and ladder poses. A clean generated trajectory does not remove calibration, state-estimation, controller, or hardware errors.

What a reviewable contact-rich data package should preserve

The site’s broader guide to human motion for humanoid training explains why source motion, robot reference, and measured execution are different records. VICAR adds a sharper requirement: the contacts and scene geometry that make an interactive motion meaningful have to remain traceable through every transformation.

For one under-table pickup, a reviewable package would preserve:

• the source video, timestamps, camera assumptions, and recovered human motion;
• each manually selected contact frame, body link, position, orientation, annotator decision, and task coordinate frame;
• depth, segmentation, scale estimate, selected obstacle regions, and voxelised collision geometry;
• the initial and refined robot trajectories, robot model, joint limits, optimiser configuration, contact residuals, clearance violations, and rejected variants;
• every augmented object or contact pose and its lineage back to the source skill;
• simulator version, policy observations and actions, randomisation ranges, reward, termination rule, and outcome;
• deployment calibration, external motion-capture stream, commanded reference, measured robot state, intervention, and hardware result.

That lineage makes it possible to distinguish an error in the video estimate from a bad contact annotation, a geometric retargeting failure, a tracking failure, or a physical interaction the representation never modelled. It also protects buyers from a misleading “hours of data” total that mixes source footage with inexpensive variations of the same processed skill.

The closest existing site analyses answer adjacent questions. GRAIL starts from specified 3D assets and video priors to generate synthetic interaction data. VICAR starts from a casual third-person demonstration and asks which world-frame contacts must survive embodiment transfer. The existing human-motion guide covers video-to-motion grounding broadly; VICAR supplies the missing contact-specific case, including movable task anchors and a hardware failure that separates a valid reference from a robust deployed skill.

The useful supervision is the interaction, not just the motion

VICAR supports a precise conclusion: third-person video can seed contact-rich humanoid learning when the pipeline reconstructs enough of the scene, labels task-defining contacts, retargets the full trajectory under robot constraints, and trains execution in physics. The reported hardware trials show that this recipe can reach a real G1 for tennis and pickup tasks without real-robot training episodes.

They do not show that raw video alone supplies robot actions, forces, reliable geometry, or deployable perception. Manual contact annotation, simulation, domain randomisation, an external motion-capture system, and task-specific engineering remain inside the result. For data teams, that is the practical lesson: package the contact, coordinate frame, scene, transformation history, and measured execution with the footage. Otherwise the most important part of the demonstration disappears exactly where the robot touches the world.