How Much History Should a Robot Policy Remember?

How much recent video should a robot policy remember? There is no universal number. Long-WAM shows that a longer observation horizon helps when the visual model was first trained to predict forward through time, the dataset contains useful uninterrupted history, and inference still finishes before the next action handoff.

The paper's most informative comparison is not simply short versus long memory. On RoboCasa GR-1, the robot-domain autoregressive variant rises from 66.3% reported success with 2.4 seconds of history to 78.7% with 19.2 seconds, while a bidirectionally pretrained initialization ends almost where it began. At 38.4 seconds, performance falls as most sampled history becomes padding. More context is therefore a resource to budget, not a knob that can be turned up indefinitely.

A single image hides direction and progress

A still frame can show a cup on a conveyor but not whether it is approaching the gripper or moving away. It can show a hand beside a bowl but not whether the robot has just placed the bowl, is about to lift it, or is recovering from a miss. Recent frames supply velocity, contact transitions, and task progress that the current image does not contain.

This is different from remembering an instruction or a shopping list. Long-WAM studies short physical history inside the low-level executor. A higher-level planner may preserve the semantic goal over minutes, while the policy needs the last few seconds to infer which way an object moved and which part of an interaction has already happened.

The distinction also separates this question from Humanoid Horizon's stage handoffs. Humanoid Horizon changes reinforcement-learning starts and rewards so one policy can complete several transport stages. Long-WAM changes the policy's visual evidence and compute path so a current action can depend on earlier observations. One addresses which states training reaches; the other asks what history the policy can use when acting.

Access to history is not the same as learning from it

Long-WAM compares three video initializations before adapting each one into the same causal world-action policy: robot-domain LongLive2.0-Robot, general LongLive-2.0, and the bidirectionally pretrained Wan2.2. An autoregressive model learns to predict later sequence blocks from earlier ones. A bidirectional video model can use information on both sides of a masked region during pretraining, even though the deployed policy is later given a causal mask.

That pretraining difference persists after adaptation. The paper reports that the bidirectional initialization moves from 61.7% success at 2.4 seconds to 64.1% at 9.6 seconds, then returns to 61.6% at 19.2 seconds on RoboCasa GR-1. The two autoregressive initializations improve by 12.4 and 11.0 percentage points between 2.4 and 19.2 seconds.

All variants can technically read the longer input. Only the autoregressive variants were previously trained on the same kind of history-to-future dependency. The result supports a narrower claim than “memory solves long-horizon robotics”: useful context depends on a representation trained to extract predictive structure from that context.

There is a reporting wrinkle worth preserving. The paper's context-ablation text and project-page abstract give 66.3% for the robot-domain model at 2.4 seconds, while Table 4 labels a 2.4-second Long-WAM row as 63.3%. The arXiv abstract describes 63.3% as the zero-history result. The shared endpoint—78.7% at 19.2 seconds—and the stated 12.4-point gain agree with 66.3%, but this article does not silently resolve the source's inconsistent label.

Video prediction becomes an intermediate control signal

The video backbone is not asked to render a polished movie before every action. Long-WAM predicts partially denoised future latents from the observed history and language instruction. An action expert then reads the history, those predicted latents, and robot state while denoising an action chunk. The future remains in latent space; it is not decoded into pixels during control.

This creates a useful division of labour. The video expert asks how the interaction is likely to evolve. The action expert asks which commands fit that evolution and the robot's present configuration. Because video pretraining needs no common action space, the authors combine video from RoVid-X, AgiBot World, EgoDex, EgoVerse, and VITRA before learning embodiment-specific actions downstream.

The headline “approximately 10,000 hours” needs careful interpretation. The appendix says the training split contains 2,251,021 samples and obtains 10,004.5 window-equivalent hours by assigning 16 seconds to each sample. It explicitly says this is not unique raw footage: overlapping windows, source clip lengths, and padding affect the total. A buyer should therefore ask for unique source duration and parent-clip lineage, not treat window-equivalent hours as independent experience.

Memory has to arrive before the action deadline

Longer history increases attention, cache, and visual-encoding work. If the policy finishes late, it may produce a well-reasoned command for a state that no longer exists. Long-WAM addresses that control latency with three connected mechanisms.

First, a streaming VAE encodes groups of observations as they arrive instead of rebuilding the entire visual window after the inference trigger. Second, asynchronous execution lets the robot consume the current action chunk while the next one is computed. Third, quantization, CUDA Graph replay, compilation, and reuse of unchanged key-value features reduce repeated work.

The authors report 107.4 ms end-to-end latency per action chunk on an RTX 5090 for the optimized four-step video/four-step action configuration, compared with 356.0 ms for their BF16 eager baseline. Reported RoboTwin 2.0 success changes from 94.4% for the eager configuration to 93.5% for the optimized one. Those figures are hardware- and configuration-specific, not a general latency guarantee.

Asynchronous handoffs add another failure surface. Two consecutive action chunks can disagree about the same future timestep. On RoboTwin 2.0, Long-WAM's reported success changes from 94.4% synchronously to 94.2% asynchronously, while its overlap error is lower than the compared Fast-WAM and LingBot-VA runs. This supports the authors' deployment design, but the benchmark does not establish safety under arbitrary delays or network jitter.

Moving objects make recent history measurable

The real-robot tests make the value of motion history easier to see than a static household benchmark. A Unitree G1 must intercept cups on a conveyor moving from 3.0 to 7.5 centimetres per second. The authors report Long-WAM success of 100%, 100%, 95%, and 90% across those four speeds, with 20 trials per condition. Fast-WAM and π0.5 both fall to zero at the two highest speeds.

The composite stacking task is more revealing than a grasp alone. The policy has to retain which cup it already holds, estimate the moving cup's trajectory, intercept it, and align the pair. Long-WAM succeeds in 19 of 20 reported trials; neither compared baseline succeeds in 20. These remain author-run evaluations on one setup, and the paper does not provide confidence intervals or an independent replication.

Longer tasks appear on a YAM bimanual robot. The paper reports 80%, 80%, and 85% success over 20 trials each for sorting bricks by colour, placing dumplings in a pan, and stacking bowls. The tasks last more than 40 seconds on average, but this does not mean the low-level model retains every frame from the whole episode. The authors argue for a shorter physical context in the executor and semantic continuity in a separate planner.

Longer-context policies need longer-context data

A context-capable model cannot learn useful history from shuffled frames or episodes that reset before consequences unfold. For data collection and procurement, Long-WAM changes the questions worth asking:

• Continuity: Are long sequences genuine uninterrupted interactions, or overlapping windows cut from the same short clips?
• Time meaning: Are image, robot-state, and action timestamps aligned closely enough for motion and contact changes to be causal rather than accidental? The checks in camera and action timing apply before context length is increased.
• Episode state: Do records preserve handoffs, failed grasps, retries, object motion, and completed substeps, or only clean clips around the final action?
• Padding and masks: How much of each sampled window is real observation, and can training distinguish repeated padding from a stationary scene?
• Embodiment lineage: Which clips are human egocentric video, which are robot video, and which include the action and state labels needed for downstream control?
• Deployment budget: What is the measured end-to-end latency at each context length on the target computer, including image encoding and handoff scheduling?

The paper's 38.4-second result shows why these questions belong together. Its average training trajectory is 12.1 seconds, so 80.4% of frames sampled for the longest window are padding. Success drops to the mid-70s. The authors do not claim a 19.2-second biological or architectural optimum; they identify a point where their current data and compute stop rewarding more history.

What the release proves—and what remains open

The public repository includes code, configuration, checkpoint download paths, and deployment interfaces. It also states that full benchmark reproduction remains unverified and documents simulator integration issues. The Hugging Face collection exposes separate RoboCasa GR-1 checkpoints for five history lengths up to 19.2 seconds, which makes the ablation more inspectable than a result with no released model.

Several limits remain:

• Each context length is a separately trained model; the study does not yet show one policy adapting its history at runtime.
• The systematic context-length sweep is in simulation. Real-robot experiments support dynamic and long-duration execution, but do not isolate context length on hardware.
• The reported gains are the authors' evaluations, and the public 2.4-second baseline label is inconsistent across the paper's text and table.
• Longer history increases compute and can repeat correlated or padded frames rather than add evidence.
• Video-only pretraining crosses action spaces, but transfer to a new embodiment still requires downstream action learning and evaluation.

The practical answer is therefore conditional. Keep enough recent observation to expose motion, contact change, and local task progress; train the representation to predict forward from that history; measure how much of the window is real; and include latency in the model comparison. Long-WAM makes 19.2 seconds look valuable for one simulated humanoid benchmark. Its broader contribution is showing why the useful memory length belongs to the data, model, and control system together.