Can Human Egocentric Data Train Different Humanoid Controllers?

Raw first-person video cannot simply be relabelled as humanoid training data. The camera is attached to a person rather than the robot, sparse human motion does not define a dynamically feasible whole-body trajectory, and each whole-body controller expects its own state and action representation. A useful conversion must align all three without losing which signals were measured, reconstructed, rendered, or optimised.

KODA proposes that conversion. It turns an egocentric RGB-D episode into a simulator-checked Unitree G1 trajectory, synthesises onboard views from the generated body motion, and then derives controller-specific state-action labels. The same canonical motion and observations are reused for SONIC and HEFT. In the authors' four-task hardware study, converted human demonstrations improve policies trained with a fixed robot-data budget—but they supplement 50 robot demonstrations per task, not replace them.

The reusable object is a canonical record, not the raw clip

Human and robot demonstrations differ in more than body shape. The source episode has a human camera trajectory, human hands, and a task instruction. A deployed humanoid policy needs the images its own camera would have observed, the state available from its robot and controller, and actions with the right dimensions, units, timing, and semantics.

KODA separates the parts that can be retained from those that must be regenerated. Its canonical record contains a robot-view image sequence, a whole-body robot configuration trajectory, and the task instruction. It deliberately excludes controller-specific actions. When a target controller is chosen, the fixed canonical motion is tracked in simulation and the realised states and compatible commands are recorded around it.

That makes this a more specific question than whether human motion can become humanoid training data. It also starts after the capture question studied by EgoExplorer. KODA asks whether the derived motion, visual observations, and action interface can remain mutually consistent when the same human source is used for different controllers.

This is a form of cross-embodiment data, but the boundary exists twice: first between the person and the G1, then between the canonical G1 trajectory and each controller's interface. A shared file format alone would not resolve either mismatch.

Motion, view, and command can disagree in three different ways

The pipeline is easier to evaluate when its alignments are kept separate:

An error in one boundary is not repaired automatically by the next. A plausible robot view can be paired with infeasible motion. A feasible trajectory can still be labelled with commands that a controller tracks badly. An accurate command can remain visually misaligned if its image was rendered from a fixed camera offset.

A simulator-checked trajectory anchors the conversion

KODA begins with RGB, camera poses, hand references, and metric depth recorded or estimated in one coordinate system. HAWOR estimates world-space MANO hands; the hands are retargeted to the robot; a fine-tuned sparse whole-body tracker converts head and wrist targets into an initial G1 reference. That first result is still only a proposal.

The next stage uses the cross-entropy method to search over corrections to pelvis and body-joint commands. At every replanning step, the paper samples candidate command sequences, simulates them, ranks their tracking costs, and refits a diagonal proposal distribution to the lowest-cost elite set. Both motion generation and later controller-command generation run at 50 Hz with 80 candidates, eight search iterations, and a 40-step horizon, although their elite counts and executed prefixes differ.

This is optimisation against a fixed kinematic target, not proof that the reconstructed human motion was measured correctly. The simulator checks whether a controller can realise a nearby motion under the chosen model and cost. The paper reports that wrist-position error falls from 15.02 cm for direct sparse tracking to 3.03 cm after this search and 0.33 cm after arm inverse-kinematics refinement. Those are source-reported errors against the derived reference, not independent ground truth about the human hand or real contact.

The distinction matters for provenance. The canonical trajectory is neither raw human measurement nor executed hardware motion. It is a derived robot reference that survived a declared sequence of estimation, retargeting, simulation, search, and refinement.

The robot view follows the generated body

Changing the motion without changing the camera creates a subtle label error. If the robot bends, turns, or takes a different path from the person, a fixed offset from the human camera can show pixels that no longer correspond to the robot state stored at that timestep.

KODA uses novel-view synthesis tied to the generated trajectory. It removes human arms, projects the RGB-D scene into a point cloud, moves the virtual camera with the generated robot, renders the G1, composites scene and robot by depth, and inpaints holes. The resulting frame is indexed on the same control clock as the canonical configuration.

The authors' ablation gives this alignment a measurable training consequence. On SONIC, macro-average success for the paper's Robot-OOD conditions is 18.75% with original human images, 23.75% with a fixed camera offset, 37.5% when reprojection follows the generated motion, and 48.75% for the full pipeline with arm removal and robot compositing. The comparison holds demonstrations, state-action labels, model, training budget, and evaluation protocol fixed.

The ablation supports training utility under this protocol. It does not independently validate pixel geometry, fill in unobserved object surfaces, or prove that inpainted regions are physically correct. The source depth and camera poses must already share metric scale, and the paper identifies fine-grained hand-object reconstruction as a remaining limitation.

One record becomes two different action interfaces

A vision-language-action model ultimately predicts the action expected by its deployment stack. KODA therefore does not label the canonical trajectory once and call it universal. It runs controller-specific command generation around the same trajectory.

SONIC encodes the optimised body reference into a 64-dimensional motion token and appends 12 hand targets, producing a 76-dimensional action. HEFT uses 29 body-joint targets plus a nine-dimensional pelvis code and the same 12 hand targets, producing a 50-dimensional action. Both training formats also include a 44-dimensional state containing 29 body joints, gravity expressed in the base frame, and 12 estimated hand joints.

The action vectors are not interchangeable even though the intended motion is shared. KODA's second sampling search finds commands that each closed-loop controller tracks more faithfully. In the SONIC ablation, omitting that search and directly using the canonical reference yields 48.75% Robot-ID and 7.5% Robot-OOD macro-average success. With controller-specific optimisation, the reported values are 72.5% and 48.75%.

This is the central reusable-data result: the canonical record reduces repeated work, but reuse still requires an adapter that understands the target controller. The pipeline avoids baking one controller into the human conversion; it does not create a controller-free action label.

The reported gains cover conditions present in the human clips

The hardware study uses four tasks on one Unitree G1 with BrainCo hands: cup to basket, parcel to shelf, basket to shelf, and cup to drawer. Each controller configuration keeps 50 real-robot demonstrations per task fixed and adds 0, 25, 50, 75, or 100 converted human demonstrations. The paper reports one policy-training run per configuration and 20 real-robot trials per task and condition.

The name Robot-OOD needs qualification. Its object appearance or pose is absent from the robot demonstrations but present in the human training pool. For example, the human clips include the replacement bottle, parcel, or tilted basket handle later used in evaluation. This measures whether converted human data can add coverage missing from the robot dataset. It is not zero-shot generalisation to a condition absent from all training data.

The upward aggregate trend appears under both interfaces, which is useful evidence for cross-controller reuse. Individual tasks and Robot-ID results are not monotonic at every intermediate budget, and one training run per setting does not reveal training variance. The 20 hardware trials estimate performance for these selected objects, tasks, and initial conditions; they do not establish broad household reliability.

The hand labels expose the remaining physical gap

KODA's body trajectory and camera alignment are detailed, but pose does not specify grasp force. The paper thresholds mean finger curvature from the retargeted hand. A grasp state switches the target to a calibrated BrainCo closure pose; a release keeps the current retargeted joint angles. The resulting labels describe a commanded hand configuration, not measured contact wrench, tactile state, or slip.

The authors state that fine-grained hand-object interactions are not yet reproduced reliably. They propose richer real-to-sim scene reconstruction and contact modelling as future work. That limitation helps interpret the four tasks: their successful placements show useful co-training under a bounded protocol, but the pipeline has not established general recovery from delicate contact, deformable objects, occluded grasps, or force-dependent manipulation.

The release boundary is similarly important. As reviewed on 17 October 2026, the official repository contains a project README and a staged release plan. The core conversion pipeline is marked in progress; command-generation code, the dataset, training configurations, and licence are still planned. The paper and media make the method inspectable, but the full result is not yet reproducible from public artifacts.

A reviewable converted episode needs both sides of every alignment

A data team evaluating this kind of pipeline should retain more than the final robot-looking video. Useful provenance includes:

• source RGB and depth, camera intrinsics and poses, hand estimates, task language, timestamps, calibration, and whether depth was sensed or inferred;
• the robot model, target embodiment, coordinate frames, raw retargeted motion, tracker checkpoint, simulator, rejection rules, and every optimisation objective and seed;
• canonical configuration and camera trajectories before controller-specific processing, including which frames were removed, reprojected, rendered, composited, or inpainted;
• controller name and version, action representation, command dimensions, state definition, control frequency, search settings, simulated rollouts, and tracking residuals;
• the real-robot demonstrations mixed into training, sampling ratio, policy checkpoint, evaluation object split, individual trial outcomes, falls, manipulation failures, and intervention logs.

Those records separate a reusable conversion from an attractive derived clip. They also make it possible to regenerate labels when a controller changes without pretending the old actions remain valid.

KODA's answer is therefore conditional but valuable. Human egocentric demonstrations can add coverage to more than one humanoid controller when a canonical robot motion and view are preserved separately from controller-specific labels. The evidence does not show robot-free training or universal transfer. It shows that observation alignment and command alignment are independent engineering problems—and that solving both can turn the same bounded human evidence into useful supervision for two different whole-body interfaces.