LeRobot v3, RLDS, HDF5, and Zarr: How to Choose a Robot Dataset Format

Choose LeRobot v3 when you want a ready-made multimodal robot-learning format, PyTorch loaders, and distribution through the Hugging Face Hub. Choose RLDS when your pipeline already uses TensorFlow and needs explicit episode and step semantics for reinforcement or imitation learning. Choose HDF5 when you want a compact, self-contained local artifact and are willing to define the robotics schema. Choose Zarr when chunked arrays must be read from file systems or object storage and your team can own the schema and chunk layout.

That answer needs one qualification: these are not four interchangeable formats at the same layer. LeRobot and RLDS tell software what a robot episode means. HDF5 and Zarr primarily tell software how arrays and metadata are stored. A team can design an RLDS-like episode model inside HDF5 or Zarr; LeRobot v3 itself combines Parquet, MP4, JSON, and Parquet metadata rather than inventing one new container.

The practical choice is therefore two choices: select the data model that preserves the meaning of an episode, then select or accept the storage layout that meets the access pattern. Confusing those layers is how a technically valid archive becomes unusable training data.

First choose a robot data model, then a storage format

A storage format answers questions such as:

• How are arrays divided into chunks?
• Can a slice be read without loading the whole dataset?
• Which compression filters are available?
• Does the data live in one file, a directory of objects, or a remote store?

A robot data model answers different questions:

• Where does one trajectory, or episode, begin and end?
• Does action a[t] lead from observation o[t] to o[t + 1], or is it aligned differently?
• Was the last step terminal, truncated by a time limit, aborted, or simply incomplete?
• Which camera frame corresponds to each robot state and command?
• What are the action space, units, coordinate frames, robot embodiment, task, and outcome?

LeRobot v3 supplies opinions at both layers. It defines feature names and episode metadata, then stores low-dimensional signals in Parquet, camera streams in MP4, and metadata in JSON and Parquet. RLDS concentrates on the episode-and-step model and exposes it through a TensorFlow data ecosystem; its domain metadata remains deliberately flexible. HDF5 and Zarr provide capable storage primitives but do not know that an array is a joint command, a camera calibration, or the final observation of a failed grasp.

This distinction also explains why converting bytes is easier than converting meaning. Copying an HDF5 array into Zarr may preserve values and dtypes while losing the convention that says whether the final action is valid. A useful conversion specification starts with semantics, not file extensions.

What every robot dataset format must preserve

The data modalities used for robot training vary by project, but the minimum contract is stable. A training-ready package should preserve:

1. Episode identity and boundaries. Every episode needs a stable identifier, an ordered set of steps, and an explicit reason it ended.
2. Observation and action alignment. The package must state which action was chosen from each observation and when its effect is expected to appear. Data synchronisation cannot be reconstructed reliably from equal array lengths alone.
3. Time. Record timestamps or a documented clock and sampling convention. A nominal 30 fps value does not explain dropped camera frames, asynchronous force samples, or controller latency.
4. Shapes, dtypes, units, and frames. A seven-value vector could be joint position, Cartesian pose, torque, or an under-documented mixture. Names alone are insufficient.
5. Task and outcome. Keep the instruction, task identifier, success or failure, interventions, reset state, and useful subtask boundaries.
6. Embodiment and calibration. Name the robot, joints, end effector, sensors, camera intrinsics and extrinsics, coordinate frames, controller, and relevant firmware or model versions.
7. Provenance and release state. Preserve the collector, capture method, source dataset, license, consent constraints, split assignment, transformation history, and dataset version.

None of the four options guarantees all seven. LeRobot and RLDS reduce the amount of schema a team must invent; HDF5 and Zarr leave more of that responsibility to the publisher.

LeRobot v3 vs RLDS vs HDF5 vs Zarr

The table is a starting point, not a performance benchmark. Throughput depends on camera encoding, chunk shape, compression, network, cache, worker count, access order, and the loader implementation. “HDF5 is fast” or “Zarr scales” is not enough information to predict a training job.

LeRobot v3 is the most opinionated end-to-end choice

LeRobotDataset v3 presents one sample API over three coordinated stores:

• data/ holds frame-level state, action, timestamp, and other low-dimensional features in Parquet shards.
• videos/ holds encoded MP4 shards separated by camera key.
• meta/ defines feature shapes and dtypes, frame rate, normalization statistics, tasks, episode lengths, and offsets into the shared files.

Version 3 replaced the earlier episode-per-file pattern with larger files containing multiple episodes. Relational metadata reconstructs episode views from those files. This reduces file-system pressure at large episode counts and supports StreamingLeRobotDataset, which can consume data from the Hub without first downloading the full release.

That design is a practical fit for video-heavy imitation learning. MP4 avoids storing every RGB frame as a separate uncompressed array, while Parquet handles columns such as joint state and actions. delta_timestamps lets a loader request observation histories or future action windows in seconds, and the returned dictionaries integrate with PyTorch DataLoader.

The cost of those conveniences is commitment to LeRobot's conventions and runtime. Feature keys, camera names, frame rate, episode indices, task records, and normalization statistics must agree. Encoded video also changes the I/O trade-off: random access may require seeking and decoding around a requested frame rather than reading one independent image object.

The software version should be pinned separately from the dataset-format version. LeRobotDataset v3 entered the stable package with LeRobot v0.4.0; at this article's publication, the project's latest tagged release is v0.6.1. A dataset card should record both the LeRobot dataset version and the writer or converter version used.

The catalog shows the format at different scales. LeRobot Shirt-Folding contains bimanual camera, state, and action streams; Robotic Origami Challenge adds tactile video and fingertip wrench signals; AIRoA MoMa 5k packages more than one million primitive-action episodes. The common loader does not make those embodiments equivalent, but it makes their differences inspectable through a recognizable structure.

Use LeRobot v3 when the destination is a PyTorch robot-learning workflow or a public or gated Hugging Face release. Do not choose it merely because the source files happen to be Parquet: the value lies in the schema, metadata, video convention, and loader together.

RLDS makes episode and step meaning explicit

RLDS stands for Reinforcement Learning Datasets. It models a dataset as episodes, and each episode as an ordered dataset of steps. A step has required boundary fields—is_first and is_last—plus optional fields including observation, action, reward, discount, and is_terminal. Episode-level metadata can include an episode ID, agent ID, environment configuration, experiment ID, and an invalid flag.

The distinction between is_last and is_terminal is especially useful. A terminal step represents an end state defined by the environment. An episode can instead be last because a time limit, interruption, or collection boundary truncated it. The RLDS specification also states that action, reward, and discount are invalid on the final observation when is_last is true. A conversion that fills those fields by copying the previous command silently changes the data.

RLDS prescribes those sequential semantics without prescribing a robot's observation vector, camera names, coordinate frames, or calibration record. All steps in one dataset must have the same fields, but projects decide what those fields contain. Data are commonly loaded as tf.data.Dataset objects through TensorFlow Datasets, while EnvLogger and RLDS Creator support environment and human-interaction collection. The RLDS paper describes the broader generation, sharing, and transformation ecosystem.

Open X-Embodiment is the clearest robotics example. Its contributors normalized more than one million trajectories from many robot datasets into an RLDS-oriented ecosystem for cross-embodiment training. The difficult work was not only serialization: contributors had to reconcile observations, actions, language, embodiments, and task definitions without pretending every source robot had the same body.

Choose RLDS when episode semantics and TensorFlow data transformations are central to the existing pipeline. It is less direct when the team expects LeRobot-compatible PyTorch loaders or wants LeRobot's prescribed Parquet-plus-MP4 distribution layout.

HDF5 is a container, not a robot schema

HDF5 stores named datasets and groups in a hierarchical file. In h5py, a dataset behaves much like a homogeneous NumPy array with a fixed dtype and rectangular shape. HDF5 dataset storage can be contiguous or chunked; chunking enables resizing and compression filters, and NumPy-style selections map to HDF5 hyperslabs.

Those features suit robotics releases well. A project can keep groups such as /episodes/000123/observations, /actions, and /timestamps in one file, attach attributes, compress large arrays, and read selected time ranges without loading everything. MimicGen, robomimic, and NVIDIA GR00T Teleop G1 are examples of releases distributed with HDF5 data.

But the path names above are a project convention, not an HDF5 rule. HDF5 does not require an episode group, distinguish terminal from truncated, define whether actions[t] belongs beside observations[t], or specify whether camera data should be frame arrays or external video. Two .hdf5 robot datasets can require entirely different loaders.

The single-file experience is attractive for downloads, checksums, and local archival. It can also become awkward when a large file must be replaced for a small correction, when many remote workers need disjoint parts, or when the chosen chunk layout conflicts with the training access pattern. HDF5 has advanced parallel and remote-access options, but a plain file in object storage should not be assumed to behave like a purpose-designed streaming dataset.

Choose HDF5 for a compact, file-oriented release or an established local pipeline whose hierarchy is already documented and tested. Publish a schema and loader with it. “Delivered as HDF5” says almost nothing about whether another team can train from the data.

Zarr favors chunked arrays across storage backends

Zarr stores typed N-dimensional arrays as independently addressable chunks, organized into groups with metadata and attributes. Zarr-Python's storage guide supports local directories, ZIP files, in-memory stores, remote stores through fsspec, and object-store implementations. Zarr v3 can also place multiple chunks inside a shard, making read chunks and stored objects different sizes.

That separation is useful when many workers read portions of large state, depth, point-cloud, tactile, or image arrays. Chunk shape can follow the expected access pattern: time-major chunks for temporal windows, for example, or chunks that keep complete low-dimensional states together. Object stores can serve individual chunks without presenting the entire dataset as one monolithic file.

The same flexibility creates design work. Zarr's array documentation defines arrays, chunks, codecs, shards, groups, and attributes—not robot episodes. A robotics publisher still has to decide:

• whether an episode is a group, an indexed region of a global time axis, or a row in a manifest;
• how variable-length episodes are represented;
• whether camera frames are chunked arrays or separately encoded video;
• where terminal state, task, success, calibration, and provenance live;
• which writes are atomic and how a reader recognizes a complete release.

Chunking is part of the public data contract. Tiny chunks can create large metadata and request overhead; very large chunks force readers to transfer and decompress unused data. Sharding can reduce the number of stored objects, but its shape must be chosen alongside chunk shape and expected concurrency.

Choose Zarr when remote chunk access, parallel readers, or large multidimensional arrays are first-order requirements. It is not a drop-in robotics standard. Without a manifest and schema, a Zarr hierarchy is simply a well-organized set of arrays.

Mapping the same robot episode into all four choices

Consider one 30-second pick-and-place episode recorded at a nominal 30 Hz:

episode_id: pick-0042
task: "Place the red cup in the tray"
robot: example_arm_v2
observations:
  joint_position: float32 [900, 7]
  front_rgb: uint8 [900, 480, 640, 3]
actions:
  joint_target: float32 [899, 7]
timestamps:
  state: float64 [900]
  front_rgb: float64 [900]
outcome:
  last_reason: success

The unequal observation and action counts are deliberate. This example stores an initial observation, 899 commands, and the final observation produced after the last command. A format conversion must preserve that convention rather than padding a meaningless final action.

In LeRobot v3, frame-level joint state, action, timestamp, episode index, and frame index would normally occupy Parquet rows. The front camera would be an MP4 stream, while episode metadata would identify the relevant row and video ranges. The dataset schema would declare the feature shapes and dtypes. The publisher would need a documented convention for the final observation if the target LeRobot pipeline expects one action per frame.

In RLDS, pick-0042 would be an episode containing ordered steps. The first step would set is_first; the final step would set is_last and is_terminal according to the environment's definition. The final action would be invalid rather than fabricated. Robot identity, task, calibration references, and outcome could be episode or step metadata.

In HDF5, the publisher might create /episodes/pick-0042/observations/joint_position, /actions/joint_target, /timestamps/state, and related datasets, with camera frames embedded or an external MP4 path stored as metadata. HDF5 would preserve the arrays, but only the accompanying specification would explain the 900-to-899 relationship.

In Zarr, the same hierarchy could be represented as groups and chunked arrays, or the episode could occupy a range in global arrays with boundaries in a manifest. Camera frames could be time-chunked arrays for direct frame access or external encoded video for lower storage cost. Again, the chosen schema—not Zarr itself—would define the final-step rule.

This is why a “lossless conversion” claim needs two tests. The first checks bytes, shapes, dtypes, and counts. The second checks episode semantics and can only pass against a written source and target contract.

How to convert a robot dataset without changing its meaning

Treat migration as a data release, not a file-copy script.

1. Freeze and identify the source. Record the source dataset version, file checksums, loader commit, and any filters applied before conversion.
2. Write the source contract. State episode boundaries, observation-action alignment, terminal and truncation rules, clocks, units, frames, missing-value behavior, and valid ranges.
3. Define the target mapping. Map every source field to a target field, transformation, external asset, or explicit omission. Do not hide dropped metadata.
4. Keep stable episode IDs. A user should be able to trace a target sample back to the source episode and, where practical, the source step.
5. Preserve raw timing. Resampling can be a derived release, but keep original timestamps and document interpolation, dropped frames, and tolerance.
6. Version calibration and embodiment metadata. Joint order, camera parameters, coordinate transforms, units, and controller meaning belong with the release, not in private setup notes.
7. Validate representative and adversarial episodes. Compare first and final steps, failures, truncations, missing frames, shortest and longest episodes, each task, and each robot or sensor configuration.
8. Run round-trip or invariant tests. Check counts, hashes where representations are unchanged, numerical tolerances where conversion is lossy, timestamp monotonicity, and video-to-state alignment.
9. Publish the converter and report. State tool versions, known losses, exclusions, and validation results before retiring the source copy.

Encoded video deserves explicit treatment. Re-encoding RGB frames is usually lossy even when frame counts match. A migration report should identify the codec, pixel format, frame rate treatment, and whether timestamps were preserved or regenerated.

The same diligence applies when buying data. The broader humanoid robot dataset evaluation checklist covers embodiment fit, rights, provenance, samples, and integration cost beyond the storage layer.

A practical format decision

Use this sequence:

1. If collaborators or downstream models require a specific ecosystem, start there: LeRobot for its PyTorch and Hub workflow, or RLDS for TensorFlow and TFDS.
2. If no ecosystem is fixed, decide whether the team wants a prescribed robotics schema. If yes, LeRobot is the more complete off-the-shelf choice of these four.
3. If the schema is already internal and stable, choose storage from the dominant access pattern: HDF5 for compact file-oriented local delivery; Zarr for independently addressable chunks across local or remote stores.
4. Prototype with real episodes before collecting or converting at scale. Measure sequential reads, shuffled temporal windows, camera decoding, worker contention, startup time, and recovery from partial writes.
5. Publish the schema, loader, validation tests, and version information regardless of the selected format.

Teams often keep more than one representation. Raw capture may remain in a loss-minimizing internal archive, a normalized LeRobot or RLDS release may feed training, and derived analytics may live in Parquet or Zarr. Multiple forms are reasonable when lineage is explicit. Untracked copies are not.

Is LeRobot just Parquet and MP4?

No. Parquet and MP4 are its main data and video storage technologies, but LeRobot v3 also defines feature metadata, task records, episode indexing, statistics, paths, temporal-window behavior, and a loader API. Reproducing the file extensions without those contracts does not produce a valid LeRobot dataset.

Do HDF5 or Zarr define a robotics schema?

No. Both can represent a well-designed robot dataset, but neither defines observations, actions, episodes, terminal states, task labels, coordinate frames, or calibration. Those semantics must come from a published project schema or from another model layered on top.

The best robot dataset format is therefore the one that preserves meaning for the next reader while matching the real I/O workload. LeRobot and RLDS buy shared semantics and ecosystem tooling. HDF5 and Zarr buy storage control. Whichever route a team chooses, the durable asset is not the extension—it is the combination of data, schema, provenance, loader, and tests.