Can Simulation Scale Humanoid Collaboration Data?

A multi-robot collaboration dataset needs more than two independently recorded humanoid robot trajectories. When one robot hands a bottle to another or holds a door, its timing and the shared object state change what the second can do. The record has to preserve that relationship.

MATE, an arXiv preprint submitted on 22 September 2026, explores one way to collect these coordinated demonstrations: two remote operators control separate humanoids in the same physics simulation. The authors report 2,500 joint episodes across five tasks. Their appendix says the total includes 1,597 validated variants derived from 903 human-operated episodes, while the hardware test uses one learned humanoid with a human partner. That makes MATE a promising data-collection study, not evidence of two autonomous humanoids collaborating in the physical world.

A joint episode keeps the relationship visible

MATE's basic data unit is a joint episode: both humanoids are recorded under a shared temporal index alongside the evolving objects and environment. Robot-specific camera views, state, and action targets stay associated with the partner's behavior. That gives a learner a timeline of who waited, who moved first, and when an object became available for the next action.

Separate videos or trajectories can show two robots moving without showing whether their actions were coordinated. During a bottle handoff, the receiver's grasp depends on the giver's timing and the object's state. During the hospital-bed task, one humanoid opens and holds a door while the other waits and pushes the bed through. If the recordings do not share a reliable clock, those interaction points are difficult to recover later. The practical issues are similar to the ones in our guide to checking camera and action timing.

The paper says each policy receives robot-centric visual observations and proprioceptive state; the complete multi-agent state is available during collection, not as a privileged policy input. The partner still affects what the learned robot sees and can do, but the policy is not given the simulator's full state as an answer key.

MATE puts two operators in one physics scene

Each remote operator controls one humanoid through an independent VR teleoperation interface. MATE runs both robots in the same MuJoCo simulation and records their interaction together. This extends the data path in our robot teleoperation systems guide: the central collection problem is preserving the two agents' coupled behavior, not just mapping a person's motion onto one robot.

The five tasks cover different kinds of coordination: opening a refrigerator for a partner, handing over or relaying a bottle, positioning a cart, and moving a hospital bed through a door. The learned robot receives its own camera view and proprioception, while the partner's role and actions shape the scene. This is useful because a model can only learn to respond to a partner if the training episodes preserve what that partner did and when.

The 24.1-hour total includes augmented variants

The preprint reports 2,500 joint episodes, or 500 per task, recorded at 50 Hz for 24.1 hours in total. Its appendix clarifies that the final set contains 903 human-operated episodes and 1,597 validated variants derived from them. The authors describe geometric changes to object positions for manipulation tasks and timing changes, such as additional waits, for the hospital-bed task. They retained variants after checking task-specific geometry, motion smoothness, contact, and replay stability.

That does not make the augmentation unhelpful. It does change what the headline means: 24.1 hours is the duration of the full dataset, not 24.1 hours of fresh, unmodified operator input. A buyer comparing datasets should be able to distinguish human-operated episodes from derived variants and trace each variant to its source. The paper's dataset statistics and construction notes make that distinction visible; a public release will determine whether it is easy to audit in the files themselves.

The hardware test is one robot with a human partner

The authors test sim-to-real transfer on Bottle Relay. A humanoid policy trained only on MATE simulation demonstrations approaches, grasps, transports, and places the bottle. During the physical trials, a human partner puts the bottle at an intermediate location with variation in position and timing; the policy receives no real-world fine-tuning. The paper reports 9 successes in 10 simulated trials and 7 in 10 physical trials.

That is an encouraging result for a narrow transfer question: can a policy trained on virtual collaboration data perform its part of a relay with a person on hardware? It does not test two autonomous humanoids coordinating with each other in the physical world. These are also author-reported results from a new arXiv preprint, with ten trials per setting, rather than an independent benchmark. The distinction matters when describing what the result establishes.

What the data release needs to make inspectable

For a buyer, the sample joint episode matters more than a single total. Can both robots be placed on one clock? Are their roles, action streams, shared object state, and contact transitions visible? Can an augmented episode be traced back to the human-operated source? Those questions show whether the data preserves the timing and provenance a policy needs to learn coordination. Our guide to robot training data modalities explains why observations, actions, state, and outcomes should remain distinct.

The MATE project page currently offers demonstration videos and says more resources will follow with the paper release. The preprint says the authors plan to release the platform, dataset, task configurations, and interaction-sampling code upon publication. A reproducible release with the joint-episode schema and augmentation history would show whether teams can inspect and reuse the reported experience.

MATE makes a strong case for treating collaboration as a synchronized data problem. Simulation gives researchers a way to collect coupled behavior without needing two physical humanoids for every demonstration. The current hardware result still covers one policy robot and a human partner in one task; the next evidence should show how much of that coordination survives when both agents act autonomously.