Robot Teleoperation Systems for Training Data: What to Compare

Choosing a teleoperation system is choosing how human intent becomes robot action data. The interface determines what the operator can control, which errors enter the demonstrations, how quickly useful episodes can be collected, and whether the resulting commands match the policy that will eventually run on the robot.

There is no universally best interface. Kinematically matched leader arms are often strongest for precise, fixed-workcell manipulation. XR controllers and hand tracking are more portable and can expose an operator to the robot's own view. Whole-body humanoid systems need a balance-aware controller between the person and the machine. Robot-free handheld systems can collect in many more places, but their trajectories are reconstructed rather than executed by a robot during capture.

The useful comparison is therefore not headset versus joystick or one published success rate versus another. It is whether the complete system preserves operator input, requested commands, controller changes, measured robot state, observations, timing, interventions, and outcomes well enough to create reproducible demonstrations.

A teleoperation system is a data path

The handheld device is only the first part of the system. A complete teleoperation path usually contains:

1. An operator interface such as leader arms, XR controllers, tracked hands, a joystick, a haptic master, or body trackers.
2. A mapping from human input into the robot's action space.
3. Inverse kinematics, motion retargeting, filtering, clipping, planning, or a learned controller.
4. The robot's low-level controller, safety limits, actuators, and physical response.
5. Visual, audio, force, tactile, and proprioceptive feedback to the operator.
6. A recorder that places observations, commands, robot state, annotations, and outcomes on a shared timeline.

Every stage can change the data. An operator may request a wrist pose that inverse kinematics modifies. A safety layer may clip the target. A whole-body controller may move the torso and feet to keep balance. The physical robot may then lag or fail to reach the command.

For policy training, those are different signals. A dataset that saves only the final target hides how much of the behaviour came from the operator, the retargeter, the controller, and the hardware. The broader guide to robot training data modalities explains why measured state and executed motion should not be collapsed into the action label.

Teleoperation architectures at a glance

These classes are not mutually exclusive. A humanoid stack may combine an XR headset, hand tracking, foot trackers, a whole-body controller, and force-aware robot hands. The table identifies where the human command enters and which transformation dominates the resulting data.

Leader–follower arms prioritise action correspondence

Leader–follower systems give the operator a smaller or passive mechanism whose joints correspond to the controlled robot. This can make joint limits, self-collision, and bimanual geometry easier to understand than a free-floating Cartesian controller.

ALOHA is a prominent low-cost example. Its paired leader arms command two six-degree-of-freedom follower arms while the system records camera views and robot data. The original work showed that a relatively inexpensive setup could collect demonstrations for fine bimanual tasks. ALOHA 2 then redesigned the grippers, gravity compensation, frame, and cameras with larger-scale collection, operator ergonomics, robustness, and maintenance in mind.

GELLO applies the same principle to several robot arms through a small, kinematically equivalent replica of the target arm. Its authors report a bill of materials below $300 for the controller itself. In a 12-person, five-task study on a dual-UR5 workcell, GELLO averaged 92% task success, compared with 72% for VR controllers and 63% for 3D mice. Those are controlled, task-specific research results rather than a universal ranking, but they show why interface geometry matters for data throughput and failure rate.

The main advantage is not that joint-space data is always superior. It is that the operator manipulates something that exposes the target robot's kinematic structure. The main cost is specialization: a GELLO device must be designed for its target arm, and an ALOHA station does not automatically transfer to a different embodiment.

Mobile ALOHA extends the idea by coupling two leader arms to a wheeled bimanual platform. The operator moves with the robot while commanding the arms, so demonstrations preserve coordinated base and arm behaviour. That is useful for mobile manipulation, but it is not equivalent to legged humanoid whole-body control. For more on the coordination problem itself, see bimanual manipulation explained.

XR and Cartesian interfaces trade directness for flexibility

XR controllers, tracked hands, and motion controllers usually command an end-effector pose rather than a matching set of robot joints. They can be deployed without fabricating a new mechanical leader and can support remote operation, but inverse kinematics and retargeting sit between the person's motion and the saved action.

DROID shows what this approach can achieve when the robot stack is standardized. Its portable workcell combines a Franka Panda arm, Robotiq gripper, two external stereo cameras, a wrist stereo camera, and Quest 2 controllers. The current project release reports 76,000 successful trajectories, 350 hours of interaction, 564 scenes, and 50 collectors across 13 institutions. The value is not only scale: shared hardware, camera calibration, language annotation, and common recording software make cross-site data easier to interpret.

The trade-off is that a Cartesian controller can request a pose that is near a singularity, outside a joint limit, or unsafe for the full arm. A dataset should therefore preserve the controller mode, inverse-kinematics result, clipping, and measured execution rather than presenting the controller pose as if it were the motion that occurred.

Open-TeleVision adds immersive stereo feedback and hand tracking. Its published system uses Apple Vision Pro poses to control the active neck, arms, and end effectors of Unitree H1 and Fourier GR-1 humanoids while returning stereo robot-camera views in a 60 Hz loop. This aligns the operator's view more closely with the robot's observations and supports high-degree-of-freedom upper-body tasks.

It is important to keep the boundary precise: the published Open-TeleVision setup does not control the humanoids' lower bodies. Its arm inverse kinematics and hand retargeting also transform human motion before it becomes a robot target, and the paper identifies missing haptic feedback as a limitation.

Whole-body humanoid teleoperation adds a balance controller

A legged humanoid cannot safely copy human joint angles directly. Different proportions, masses, joint limits, feet, actuators, and contact conditions make visually similar poses physically different. A whole-body controller or learned motion policy must turn sparse human goals into feasible robot motion.

OmniH2O illustrates this architecture. Its VR mode uses head and hand tracking as sparse kinematic goals while a learned controller generates whole-body motion for a Unitree H1. The paper describes a 50 Hz policy above a 200 Hz proportional-derivative control loop. Its released OmniH2O-6 dataset contains six everyday tasks and records demonstrations at 30 Hz. Those three rates describe different layers; none should be substituted for an end-to-end latency measurement.

NVIDIA's current GR00T Whole-Body Control data-collection documentation provides a more explicit example for Unitree G1. The SONIC controller runs at 50 Hz and accepts full-body or sparse VR input. Its exporter writes LeRobot v2.1 datasets with measured joint positions and velocities, body orientation, projected gravity, ego and optional wrist images, target joint positions, target body rotation, task text, and teleoperator pose data.

The same stack lets an operator save or discard an episode, retains discarded recordings until post-processing, and detects stale body-pose frames caused by pauses or packet drops. Its teleoperation safety guide also requires a clear operating zone, a separate safety operator, calibration checks, and accessible emergency stops. These details do not prove that the system is right for every buyer, and the released controller targets the G1. They do show what an inspectable collection pipeline should document.

The central data question is authorship: which motion came from the person, which came from retargeting, and which was synthesized by the balance controller or planner? A whole-body dataset should retain all three layers when possible. Otherwise a buyer cannot tell whether a footstep, torso correction, or hand trajectory was demonstrated, inferred, or generated for stability.

Robot-free demonstrations are not teleoperation

Some of the most scalable collection tools deliberately remove the live robot from the loop. They belong in a buying comparison because they compete for the same collection budget, but they should not be labelled teleoperation.

Universal Manipulation Interface, or UMI, uses a handheld parallel-jaw gripper with a camera to collect demonstrations in ordinary environments. Camera and inertial data are used to reconstruct relative six-dimensional trajectories, gripper width is recovered from the device, and the resulting motion is filtered against target-robot constraints. The robot executes a learned policy later; it is not being driven during capture.

That distinction changes both the advantage and the risk. UMI can move between homes, restaurants, and other sites without moving a robot. On the cup-arrangement task, its authors report 111 demonstrations per hour, compared with 35 using SpaceMouse teleoperation and 231 by bare hand. The figures are specific to that task and experimental setup, but they show how removing robot resets and motion limits can increase collection throughput.

The cost is weaker action ground truth. A UMI trajectory comes from visual-inertial reconstruction and feasibility filtering, not measured robot execution. Buyers need the raw camera and inertial records, reconstruction method, calibration, discarded-trajectory criteria, target robot geometry, and latency assumptions used at deployment. The guide to data collection grippers covers this hardware category in more detail.

Control frequency does not measure responsiveness

Control frequency says how often a loop or recorder updates. Control latency says how long information takes to travel through a defined part of the system. A 50 Hz controller can still act on an old camera frame, and a 30 Hz dataset can be generated by a faster inner control loop.

For a teleoperation system, at least four time boundaries matter:

• Human input capture to command generation.
• Command generation to controller receipt.
• Controller receipt to measured robot response.
• Robot-camera exposure to the image shown to the operator.

Round-trip visual feedback is different again. Average delay is also insufficient because jitter and packet loss can turn occasional commands into large errors. NVIDIA's SONIC model card, for example, explicitly warns that its low-latency checkpoint's roughly 80 ms reference lookahead is not a measurement of end-to-end teleoperation latency.

A useful system report should state the clock source, timestamp location, mean and high-percentile latency, jitter, dropped-frame policy, and whether measurements include sensing and actuator response. The saved streams then need data synchronisation rules for aligning different rates without moving actions to the wrong side of a contact event.

Haptic feedback must cross the interface

A device is not haptic merely because the operator holds it. True bilateral or force-reflecting teleoperation measures interaction on the robot and returns a controlled force or resistance to the operator. Passive friction in a leader arm, controller vibration, and visual contact cues are different signals.

For contact-rich tasks, ask:

• Which robot-side force, torque, tactile, or contact measurements drive operator feedback?
• What force scale, bandwidth, saturation, and filtering are applied?
• How does the system remain stable when delay or packet loss increases?
• Are raw robot-side measurements, returned operator-side forces, and safety clipping all recorded?
• Does the training dataset retain the contact signal, or was it available only during collection?

Force reflection can help an operator feel insertion, collision, slip, or excessive load. It also adds cost, calibration, and safety complexity. If the recorded dataset contains only video and position targets, the haptic hardware may have improved collection but does not give the eventual policy access to force information.

Throughput means accepted episodes per wall-clock hour

Published control rates and raw recording hours do not reveal collection economics. A system that captures a smooth 30-second task may still need several minutes for scene reset, calibration, failed attempts, file checks, and robot recovery.

The operational metric should be accepted, usable demonstrations per wall-clock operator-hour. Track at least:

• Attempted, successful, discarded, aborted, and safety-stopped episodes.
• Setup, calibration, reset, labelling, upload, and maintenance time.
• Operator learning time, fatigue, and performance drift across a shift.
• Self-collisions, environment collisions, tracking losses, and unreachable commands.
• Diversity across operators, robots, objects, environments, and days.
• Missing frames, timing violations, calibration drift, and export failures.

ALOHA 2's emphasis on gripper ergonomics, simpler maintenance, and a more open workcell is relevant because small reductions in effort and downtime compound across a fleet. DROID demonstrates the other side of scale: standardized rigs and procedures can support distributed collection, but the hardware and calibration burden travel with every site. UMI reduces robot occupancy, but adds reconstruction and embodiment validation.

Cost comparisons must use the same boundary. GELLO's reported sub-$300 figure covers its controller parts, while ALOHA, Mobile ALOHA, and DROID involve robot arms, cameras, compute, frames, and safety equipment. Hardware price alone is not cost per accepted training hour.

Require a field-level dataset specification

Before buying a teleoperation stack or licensing its output, request a sample episode and a schema that separates these layers:

1. Embodiment: exact robot revision, joint order and limits, end effectors, sensors, firmware, controller configuration, and calibration files.
2. Human input: interface model and firmware, raw tracked poses or device state, confidence values, occlusion flags, and operator calibration.
3. Mapping: coordinate frames, scaling, inverse kinematics, retargeting version, smoothing, clipping, unreachable-pose handling, and planner contribution.
4. Commands: operator request, retargeted target, controller command, safety-modified command, and the units and reference frame for each.
5. Execution: measured joint position and velocity, base state, end-effector pose, gripper or hand state, and force or tactile channels when present.
6. Observations: raw camera streams, intrinsics, extrinsics, exposure settings, depth, audio, and exactly what the operator saw.
7. Timing: original timestamps, clock source, source rates, saved rates, measured offsets, dropped samples, interpolation, and latency boundaries.
8. Episodes: task instruction, initial state, success rule, subtask boundaries, operator and session identifiers, and environment metadata.
9. Failures: unsuccessful attempts, pauses, resets, corrections, collisions, falls, emergency stops, tracking loss, and reasons for exclusion.
10. Rights: operator consent, privacy handling, capture-location permissions, software and model restrictions, and the dataset's commercial licence.

A red flag is an episode containing only images and target actions. Without measured execution, timestamps, controller identity, and intervention records, a buyer cannot calculate tracking error, identify stale commands, or distinguish operator skill from hidden controller assistance. The broader dataset evaluation checklist covers licensing, provenance, and delivery diligence beyond the teleoperation stack.

Run a matched acceptance test

Research benchmarks are useful evidence, but they do not select a system for a different robot, workforce, and task distribution. Before standardizing hardware, run competing interfaces through the same acceptance test:

1. Define a representative set of precision, reach, contact, bimanual, mobile, and recovery tasks.
2. Define the policy's intended observations and action representation before collecting data.
3. Give operators the same training time and rotate interface order to reduce learning bias.
4. Record every attempt, reset, collision, intervention, and rejected trajectory.
5. Measure accepted episodes per wall-clock hour, not only successful task time.
6. Inspect median and high-percentile latency, synchronization error, missing data, and calibration drift.
7. Export sample episodes and train or replay a small baseline before committing to a fleet.

The selection rule is practical. Use kinematically matched leaders when fixed-workcell precision and direct action correspondence matter most. Use XR or tracked controllers when portability, remote operation, or upper-body range matters more than direct joint mapping. Use a documented balance-aware stack for legged whole-body data. Use robot-free capture when environment diversity is the bottleneck, while budgeting for reconstruction and transfer validation.

The winning system is the one that produces the most usable, inspectable data for the target policy—not the one with the most impressive demonstration video.