LIBERO vs LIBERO-Plus: What a Robot Policy Score Leaves Out

A strong robot benchmark score does not tell you what happens when someone moves the camera. In the LIBERO-Plus paper's initial robustness analysis, OpenVLA-OFT achieved 97.1% success on unperturbed inputs, 59.7% under camera perturbations and 37.2% when the robot's starting state changed. Those are results reported by the researchers, not tests conducted for this article.

That gap is the reason to evaluate a policy on both its familiar tasks and controlled variations. LIBERO provides a task framework for studying knowledge transfer. LIBERO-Plus extends it to expose weaknesses under changed observations and conditions. Neither score, by itself, establishes that a humanoid will work in a new room.

The useful question for a training-data team is more specific: which variation breaks the policy, and does additional data fix it on scenarios kept out of training? Answering that requires more than replacing one leaderboard number with another.

LIBERO and LIBERO-Plus ask different questions

The original LIBERO repository describes 130 tasks across LIBERO-Spatial, LIBERO-Object, LIBERO-Goal and LIBERO-100. The first three control changes in spatial, object or goal knowledge. LIBERO-100 combines these demands and is split into LIBERO-90 for pretraining and LIBERO-10 for downstream lifelong-learning evaluation.

So it would be wrong to say LIBERO never tests distribution shift. A distribution shift means that the conditions represented at evaluation differ from those encountered during training. LIBERO already studies particular forms of transfer; LIBERO-Plus asks how reliably policies handle additional changes around the tasks.

The LIBERO-Plus construction produces 10,030 task variants across object layout, camera viewpoint, robot initial state, language instruction, lighting, background and sensor noise. These are variations of simulated manipulation problems, not 10,030 independently demonstrated real-world skills. The researchers filter and balance the expanded tasks, then assign five difficulty levels using results from four representative models. Difficulty is therefore calibrated against those models, not a universal measure of physical difficulty.

There is an important reporting distinction inside the paper itself. The opening numbers above come from its initial single-perturbation analysis, Table 1. The later, constructed LIBERO-Plus benchmark has a separate Table 2. They should not be combined into a single before-and-after comparison. A result needs its evaluation set as well as its model name.

Our simulation-environment comparison separates physics engines from task suites. LIBERO-Plus belongs on the evaluation side of that distinction: its value is in specifying what to change and what to measure, not in replacing the simulator with a more photorealistic one.

Change the camera before buying more textures

The OpenVLA-OFT results make a useful procurement point. In that same Table 1 analysis, background perturbations left the model at 92.4% success, while changing the robot's starting state left it at 37.2%. A dataset rich in new table textures could address a much smaller weakness than a dataset with broader starting configurations.

That is an interpretation of the reported results, not evidence that textures never matter. The failure profile belongs to a particular model and setup. A team should measure its own profile before choosing what to collect or generate.

Sensor configuration changes the interpretation too. The paper compares OpenVLA-OFT with a variant lacking the wrist-camera input. Under camera perturbations, the reported scores were 59.7% with the wrist view and 16.8% without it. The visual-input ablations also show that retaining a wrist view can preserve substantial performance when the third-person image is masked.

A camera-robustness claim must therefore say which camera moved and which views remained available. A stable wrist view can help when an external camera changes. That does not prove robustness when every camera changes, or when a humanoid's head and torso move together.

For a data purchase, ask for examples spanning the failing condition, with the camera setup and starting state recorded. Domain randomisation is useful only to the extent that its chosen variations address the weakness being tested.

A paraphrase test can reward ignoring the instruction

Language provides a more surprising trap. If a vision-language-action model succeeds after an instruction is reworded, it might understand the paraphrase. It might also be performing the same visually familiar action regardless of the words.

The researchers tested that distinction by removing instructions and by changing the requested target. In the blank-instruction experiment, OpenVLA-OFT's performance on the Object suite remained largely unchanged; the Long suite degraded substantially. That is a suite-specific finding, not proof that every VLA always ignores language.

The goal-replacement experiment is more revealing. The paper gives the example of replacing “pick up the alphabet soup” with “pick up the tomato sauce”, changing both the instruction and the task goal within the scene. Success fell nearly to zero in the tested replacement scenarios, and inspected rollouts often continued towards the original target.

This suggests a practical evaluation pair. First, preserve the intended goal and rephrase the instruction: the desired action should remain the same. Then keep the scene comparable but request a different valid target: the desired action should change. Update the success condition with the target, or the evaluator will reward the wrong behaviour. These are proposed checks based on the paper, not new experimental results.

For language-labelled training data, the implication is concrete. More descriptions of the same scene-action pairing may add linguistic variety without establishing instruction following. Examples where different instructions require different actions in comparable scenes are worth investigating separately.

More varied training helps, but inspect what it omits

LIBERO-Plus also investigates a remedy. The authors generated more than 20,000 successful training trajectories and used them for mixed fine-tuning. On the constructed benchmark, Table 2 in the paper's third revision reports 79.5% overall success for their resulting model, with 92.8% under camera perturbations but only 30.3% under robot-initial-state perturbations. The improved total still contains a substantial weakness.

The detail that matters most for data selection is in Appendix D. The training set spans six kinds of variation. Its object-layout augmentation adds distractors, but excludes target-pose changes because automatically generated trajectories were not reliable enough. Only successful trajectories were retained.

That omission does not prove why a particular score stayed low. It does show why “diverse synthetic data” is too vague a product description. A generator may reliably produce new appearances around an existing action while struggling to produce the new action required by a changed configuration.

Before paying for more generated episodes, request the generator's exclusions as well as its supported variations. Which changes were attempted? Which were rejected? Does the delivered set include recovery from the states where the current policy fails, or only successful executions close to familiar paths? Episode volume cannot answer those questions.

Keep the test out of the data-improvement loop

A weak shifted score is not evidence of benchmark leakage. It can result from narrow training coverage or reliance on a shortcut. Leakage is a separate problem: information from the test influences model development, making the final evaluation less independent. The scikit-learn guidance on data leakage explicitly warns against using test data to make model choices, including during preprocessing.

The LIBERO-Plus authors do describe separation measures. Appendix D.2 says training distractors and their poses avoid test overlap, background textures avoid test environments, and training camera angles differ from test angles by five degrees in spherical coordinates. That is evidence of a designed separation, not an independent audit of every released trajectory. It also does not establish transfer to an entirely different camera arrangement.

For a team's own data-improvement study, I would make the separation explicit before generating new episodes:

1. Declare the generalisation claim. Testing new episodes in familiar scenes is different from testing unseen scenes, objects or camera placements. Define which factors must be held out, then split by those factors. Renaming a file or choosing another random seed does not establish an unseen-scene test.
2. Use a development set to diagnose failures. Inspect those rollouts, choose augmentations and compare candidate checkpoints there. Fit learned normalisation statistics on training data, not on the final test set.
3. Freeze a separate final evaluation. Keep its scenario definitions out of the collection and tuning loop. If its failures become training material, retain the old scenarios as regression tests and reserve a new untouched set for the next generalisation claim.
4. Compare on the same declared protocol. Record the checkpoint, dataset and simulator revisions, camera inputs, action representation, task IDs, starting states, episode limits and success rules. Report completions and trial counts by perturbation category, not just an average.

These are recommended controls, not additional requirements claimed by LIBERO-Plus. Even its repository's evaluation instructions matter: they specify changing num_trials_per_task from 50 to 1. Copying a familiar LIBERO command without checking the variant's protocol can produce a differently defined result. Repeated trials can help assess variability, but any departure from the published protocol should be labelled.

A manipulation benchmark is not a humanoid acceptance test

The experiments discussed here concern simulated manipulation and input perturbations. They do not validate a humanoid's balance, walking, contact safety or whole-body control. Even strong results leave a separate sim-to-real question: whether the policy works with the target hardware, sensors, controller and physical environment.

That limit does not make LIBERO-Plus irrelevant to humanoid teams. It makes the benchmark useful for a narrower decision: identifying a manipulation policy's weak conditions and checking whether a proposed data intervention improves them. The training-data evaluation guide covers the accompanying checks on embodiment fit, provenance and rights.

The evidence to ask for is a named failure condition, a documented change to the training data, and an improvement on an evaluation the training process did not consume. If the average rises while the robot still fails from a different starting pose, keep that failure in the purchasing decision.