Why Robot Grasps Work in Simulation but Fail on Hardware

A simulated grasp can succeed because the robot learned a useful skill. It can also succeed because the simulator gave the fingers more friction, a different collision shape or a more forgiving contact response than the physical robot has. Before collecting more demonstrations, check whether the successful simulated action is physically plausible.

This is a specific part of the sim-to-real problem: validating the interaction that holds an object, seats a peg or supports a foot. Better textures will not repair a collision model that fills in the hole the robot is supposed to use. NVIDIA's own Isaac Sim tutorial demonstrates exactly that failure: a cube rests above a torus's visible hole because the default convex collision approximation closes it.

The question for a training-data team is therefore not just whether a simulation episode ends in success. It is whether the geometry, contact forces and controller that produced that success are a defensible model of the target task. I would establish that before treating the episode as a good demonstration.

Inspect the shape the physics engine actually uses

A visual mesh answers what the object looks like. A collision representation answers where the simulator generates contacts. Those representations can differ deliberately: simpler collision shapes often cost less to simulate.

In the Isaac Sim torus example, switching to a decomposition made of several convex shapes can preserve the opening. The documentation also describes an SDF Mesh option for rigid bodies, using a signed distance field: a representation of distance to the object's surface. These are modelling choices, not a universal ranking in which the most detailed mesh always wins.

For a gripper, inspect the fingertip collision surfaces with the rendered hand hidden. For insertion, inspect the socket opening and the plug together. A simplified fingertip can change where the object is supported; a simplified socket can remove the valid path altogether. Increasing the friction coefficient is the wrong repair for either problem.

Even sharing the CAD source does not guarantee that two teams simulate the same asset. The IndustRealKit repository's OBJ notes explicitly say its convenient CAD exports are not the exact meshes used for policy training: the training meshes were post-processed. The authors also warn that their Factory SDF-based contact scheme is sensitive to mesh quality.

The physical parts need checking too. In the same repository's printing notes, the authors report measuring diametral clearances of 0.5–0.6 mm between mating parts after printing. They caution that different clearances can affect policy performance. That is a measurement of their parts, not a recommended tolerance for every assembly task. A mesh file and a printer setting are not substitutes for the dimensions of the delivered hardware.

Friction belongs to a contact pair

It is tempting to describe an asset as having “friction 0.5” and consider the issue settled. But the effective interaction depends on both contacting surfaces and on the engine's combination rules.

For dynamically generated contacts, MuJoCo's contact-parameter rules use the friction coefficients of the higher-priority geometry. When priorities are equal, they use the element-wise maximum. Explicitly defined contact pairs instead specify their own parameters and ignore the individual geometries' values.

That produces a useful diagnostic example. Suppose two equal-priority geometries have sliding-friction settings of 0.5 and 1.0. Under that MuJoCo rule, the contact uses 1.0, not an average. Reducing only the first geometry's value will not make the contact less grippy while the second remains at 1.0. This follows from the documented rule; it is not a simulation result measured here.

Isaac Sim's physics materials expose configurable combination modes, including average, minimum, multiplication and maximum, with precedence when the two materials disagree. Copying the same material numbers between engines can therefore produce a different effective interaction.

The friction cone is useful for reasoning about admissible contact forces, but checking a cone built from an assumed coefficient does not validate the coefficient. For a dataset, retain the material settings on both sides, the combination rule and any pair-specific override. A field containing one unexplained friction number is not enough to reproduce the grasp.

Contact tuning can conceal a geometry error

Contact settings control more than whether two shapes intersect. In Isaac Sim, rest offset can inflate or shrink the effective collision geometry. Contact offset controls how far from the collision geometry the engine begins generating contact constraints. A larger contact offset costs more computation; too small an offset can cause late or missed contacts and jitter.

Those two settings are not interchangeable. Starting contact generation early does not simply mean that the object has a physically thicker surface. Record both, and inspect whether the resting separation and the onset of interaction agree with the intended geometry.

MuJoCo exposes a different formulation. Its solver-parameter documentation describes soft constraints whose behaviour is configured through solimp and solref. Broadly, solimp controls constraint impedance, while solref controls the reference response through a time constant and damping ratio, or a direct stiffness-and-damping format. The friction directions have their own qualifications; these are not generic material stiffness fields that can be copied into another engine.

A successful rollout after making contacts harder is evidence that the rollout is sensitive to the contact settings. It is not, by itself, evidence that the revised settings are more realistic. I would compare the contact onset, resting pose, slip and release against a controlled physical reference before accepting the change.

Nor should validation require literally zero overlap in every engine. Soft-contact models intentionally permit some penetration. The relevant distinction is between the model's intended compliance and an implausible shortcut, such as an insertion that crosses a solid wall.

A policy can learn the simulator's mistake

This is not only a hypothetical concern. The IndustReal paper, Section IV-E, describes policies exploiting inaccurate collision dynamics. Its simulation-aware policy update filters or downweights training episodes using interpenetration depth, encouraging the agent to avoid solutions that depend on those errors.

The implication for generated demonstrations is straightforward. A success label should not be the only retention criterion. An episode can satisfy a goal-position check while taking a physically implausible route to that goal. Preserve the evidence needed to inspect how it succeeded: object and robot poses, collision configuration, relevant contact information and the success rule.

IndustReal also separates partial engagement from full insertion in its evaluations. That distinction matters for a data product. “The peg entered the opening” and “the peg reached the required seating depth” teach and evaluate different outcomes. A seller should not collapse them into an undocumented success flag.

These findings concern the paper's assembly setup, not humanoid balance or arbitrary household manipulation. They support a validation principle, not a claim that its particular filtering threshold should be reused for every hand, material or simulator.

Check the controller before blaming the data

A learned policy typically supplies commands to another controller. The meaning of those commands, their update rate and the resulting motion are part of the simulation-to-hardware contract.

The IndustReal deployment discussion, Sections V-A and V-C, makes this concrete. The researchers distinguish physics, low-level control and policy rates. They also report that removing arbitrarily introduced dissipative terms, such as heuristic friction and damping, partially reduced deployment error. Their response was not simply to add more training episodes.

Control frequency, physics-step frequency and saved video frame rate are different quantities. Isaac Sim's timeline documentation explicitly allows multiple physics steps per rendered frame. A smooth video can hide contact behaviour occurring between its frames, and a matching video rate says little about whether the hardware controller receives equivalent targets.

For a transfer diagnosis, keep the policy checkpoint fixed and check the action space, units, control mode, gains, command limits and timing. Record requested commands separately from measured robot state. Otherwise, a controller mismatch may be misdiagnosed as a lack of task demonstrations.

Validate a small interaction before scaling the dataset

I would start with a narrow acceptance exercise rather than an entire autonomous task. The following is a proposed workflow based on the modelling issues above, not a new benchmark or a claim of hands-on testing:

1. Choose the decisive interaction. For grasping, isolate contact, lifting, holding and release. For assembly, separate alignment, initial engagement and final seating. Define observable failure conditions before tuning anything.
2. Establish a physical reference under approved operating limits. Record the actual parts, surface condition, controller and synchronised observations. Use qualified operators and the robot's safety procedures; do not increase force limits to imitate an implausibly strong simulated grasp.
3. Compare the same interaction in simulation. Check collision geometry first, then contact-pair parameters and controller behaviour. Inspect slip, resting pose and contact onset as well as the final success flag. Where forces are measured, preserve their calibration and coordinate frame.
4. Change one modelling assumption at a time. Test whether the discrepancy is sensitive to geometry, material combination, contact settings or timing. If several settings can explain the same observation, retain that uncertainty rather than claiming a uniquely identified material value.
5. Validate on held-out interactions. A model tuned on one lift should face different starting poses or loads within the intended operating envelope. Keep a separate test after tuning; matching the calibration example alone is not transfer evidence.

Only then would I expand domain randomisation around defensible ranges. Randomisation can cover uncertainty, but it cannot recover a missing socket opening. Preserve the generator configuration, asset revisions, sampled parameters and rejected-episode reasons alongside the delivered data.

Our simulation-environment comparison helps choose the software stack. The more consequential purchasing question comes afterwards: can the supplier show why the contacts in a successful episode should work on the target hardware? Until that is demonstrated, a larger archive may simply contain more examples of the same modelling error.