Can a Phone Scan Personalize a Robot Policy?

Can a phone scan replace robot demonstrations in every new home or office? Splat2Act reports a qualified yes for receptacle-local pickup: a 20–30 minute RGB-D scene capture, one image and the dimensions of each chosen object, and an automated simulation pipeline produced the data used to adapt a vision-language-action policy. No target-scene teleoperation was used.

The qualification matters. The authors tested a Stretch 3 mobile manipulator in two captured campus scenes, not a general household robot across arbitrary buildings. Their strongest scene-matched policy reached 62.5% success in its simulated Office task bank and 63.9% on a 12-task real-robot subset. The pipeline replaces one kind of labour—repeated target-environment demonstrations—with reconstruction, simulation, oracle planning, and validation. It does not make deployment data-free.

The capture is an input, not the finished simulator

Splat2Act starts with an iPhone RGB-D sequence captured through Polycam. It trains a 3D Gaussian splatting representation, regularised with depth and surface normals, to render camera observations that resemble the recorded space. It then extracts a mesh with SuGaR. The two representations do different jobs: the Gaussian scene supplies appearance, while the mesh supplies collision geometry, surface discovery, and navigation constraints.

That split is easy to miss. A visually persuasive reconstruction is not automatically safe for contact or planning. The real-to-sim record needs the source frames and camera poses, the trained Gaussian representation, the extracted mesh, scale and coordinate alignment, and the reconstruction settings. If only the rendered video survives, a later team cannot tell whether a failure came from the learned policy, a missing surface, or an incorrectly scaled collision model.

The seven personalised objects are captured separately. Splat2Act uses SAM 3D to construct a mesh from one image, then scales it to dimensions supplied by the user. The paper explicitly says these are object-specific assets without assuming exact reconstruction. Shape, texture, mass, friction, and graspability therefore remain separate questions; one photograph plus dimensions does not recover all of them.

Geometry turns a room into pickup tasks

The system does more than reconstruct a backdrop. It searches the mesh for upward-facing regions with usable heights and areas, clusters surface elements, rejects unstable or badly shaped regions, and samples candidate object placements. A Habitat navigation mesh then filters out placements the mobile base cannot approach. Settlement, grasp, and demonstration-execution checks remove more infeasible cases.

That pipeline found 852 retained receptacles across 101 campus scenes with a median navigable area of 104.87 square metres and seven receptacles per scene. Receptacle means a usable support surface here, not only a table: the geometric search can retain shelves, counters, and other reachable horizontal regions. Its thresholds are tied to the Stretch geometry, so the same scan would not automatically yield the same task set for a taller humanoid, a wider base, or a two-handed reach.

The demonstrator can see what the deployed policy cannot

Successful trajectories come from a scripted simulation oracle. It reads exact object poses, geometry, and robot state; ranks grasp and base configurations; executes pregrasp, descent, closure, and lift phases; and restores a simulator snapshot to try another candidate after failure. This is privileged information: useful during data generation, but unavailable to the deployed policy, which receives head and wrist RGB, proprioception, and a language prompt.

The boundary should be explicit in any derivative dataset. Oracle object poses and success predicates can label or generate episodes, but they must not quietly enter the deployed observation vector. Otherwise the result measures a simulator-assisted policy rather than the camera-driven policy a real robot can execute.

At 64 parallel environments, the authors report an average of 60 successful pickup demonstrations per A40 GPU-hour. Physics runs at 100 Hz; control and camera recording run at 20 Hz. Those figures describe this simulator, task family, and hardware. They are not a general price estimate for generating robot data.

Sixty-eight thousand trajectories are not one training set

The two broad generation campaigns recorded 68,021 successful trajectories, representing 273.46 control hours. One campaign remained in the Office scene; the other spanned all 101 scenes and 556 compatible Objaverse assets. The paper also describes a much larger combinatorial space—up to 473,712 object–receptacle pairs and roughly 11.4 million tasks if every pair received 24 feasible configurations—but those are potential counts, not collected demonstrations. The multi-scene campaign was subsampled by about 100 because of storage limits.

Even the 68,021 figure is not the number used for each policy. The first adaptation stage samples 1,024 trajectories for Office1k and another 1,024 for Multi1k. The second, personalised stage uses 504 Office demonstrations or 500 Conference demonstrations. This separation is valuable: generation capacity, stored corpus size, and consumed training set are three different quantities.

It also reveals selection bias. The retained training corpus contains successful oracle demonstrations. The paper reports 79.18% oracle generation success for broad Office and 65.32% for the broad multi-scene campaign, so thousands of assigned attempts did not become successful trajectories. A useful dataset card should preserve failed assignments and rejection reasons alongside the successful episodes; otherwise users cannot see which objects, surfaces, or grasps the generator systematically avoided.

Personalisation works best when the scene matches

Splat2Act uses two supervised fine-tuning stages. Stage 1 adapts the pretrained π0.5 model to the Stretch embodiment, action space, objects, and receptacles using either Office1k or Multi1k. Stage 2 fine-tunes on the user objects in one deployment scene. The image encoder stays frozen while the language backbone and action expert are updated with the model's conditional flow-matching objective.

The strongest Office result follows same-scene broad adaptation: Office1k then Office personalisation reaches 62.5% in simulation, compared with 45.0% for direct personalisation from the pretrained model and 47.5% after Multi1k. In Conference, Multi1k then Conference reaches 51.4%, compared with 42.9% from the pretrained model and 37.1% after Office1k. The pattern is not “more scenes always wins”. Scene-matched data gives the best in-scene result, while diverse multi-scene data provides the stronger starting point in a different scene.

The pretrained π0.5 checkpoint scores 0% in both task banks, but the authors attribute that to missing exposure to the Stretch embodiment and their action space as well as to the scenes. It is not evidence that generalist VLAs universally score zero in unfamiliar rooms.

Wrong-scene personalisation can be actively harmful. An Office1k policy scores 40.0% on Office before Stage 2, but only 15.0% there after being personalised to Conference. Fine-tuning on familiar objects in the wrong visual and geometric context can overwrite useful behaviour instead of adding object knowledge.

Augmentation depth grows with the task surface

The paper separately varies receptacle count and the number of tasks generated per object. With four rendered objects, eight tasks per object gives success between 21.9% and 43.8% across the tested receptacle counts. At 24 tasks per object, success is 75.0% with one receptacle, 87.5% with two, and 75.0% with four.

These are in-distribution simulation ablations, not a universal rule that every object–surface pair needs exactly 24 demonstrations. They do show why “number of objects” is an incomplete coverage metric. Robot pose, object placement, distractors, reach direction, and support geometry create distinct interaction configurations even when the object label stays the same.

For buyers of synthetic robot training data, the practical unit is therefore a configuration lineage: object asset, receptacle region, robot start, target pose, distractors, generator version, oracle result, and whether the configuration entered training or remained held out.

The real-robot result is encouraging and small

The real evaluation uses two Office-personalised policies on 12 tasks across three receptacles. Each policy receives one simulated episode per task and three real attempts per task: 12 simulation episodes and 36 real trials. The Office1k-based policy reports 66.7% in simulation and 63.9% on Stretch 3. The Multi1k-based policy reports 50.0% and 58.3% respectively.

The authors observed less precise arm extension and grasp placement on hardware, while Stretch's compliant gripper and rubber fingertips sometimes made real contact more forgiving than the simulated model. That is a useful warning against assuming the simulator is always optimistic. Similar aggregate success can hide different failure mechanisms.

The unseen-scene tests show the other side of the result. On two new lab tables, the best reported success lies between 22.2% and 33.3%, far below the scene-matched Office results. Each table uses 18 trials per policy. The ordering also reverses between tables, so the evidence does not establish one consistently better adaptation route outside the personalised environment.

What the dataset must preserve

A reproducible Splat2Act-style package needs more than successful action arrays. I would expect:

• original RGB-D capture frames, camera intrinsics and poses, capture timestamps, processing versions, and reconstruction failures;
• the Gaussian representation, extracted collision mesh, coordinate transform, navigation mesh, discovered and rejected receptacle regions, and reachability thresholds;
• object source images, user-supplied dimensions, generated meshes, scaling rules, rights, and any physical properties added by the simulator;
• every task blueprint, robot and object initial state, distractor set, oracle retry count, failed grasp candidates, success predicate, and simulator version;
• a strict split between simulator-only privileged fields and observations available to the deployed policy;
• 20 Hz image, state, and action timing, action-chunk boundaries, training subset membership, held-out configurations, and real-robot outcome logs.

This is also where Splat2Act differs from the existing SimFoundry analysis. SimFoundry asks whether reconstructed worlds preserve policy evaluation and support digital cousins. Splat2Act asks how automatically discovered, scene-specific demonstrations should be staged for policy personalisation, and shows that the source of Stage 1 data changes both in-scene performance and transfer.

The public release is not yet reproducible

The paper was submitted on 9 October 2026. Its project page currently marks both code and dataset as “coming soon”, while the paper says the 68,021-demonstration dataset will be released upon acceptance. I found no public repository or dataset licence linked from the official page during this review. The figures, method description, and videos are unusually detailed, but they do not yet let an independent team rerun the pipeline.

The scope is also narrower than the headline may suggest. The evidence covers receptacle-local pickup, two target deployment scenes, three personalised target identities, and one Stretch 3 embodiment. It does not test whole-body humanoid reach, stairs, doors, deformable objects, long-horizon fetching, safety around people, or continuous adaptation after a room changes.

Splat2Act nevertheless makes a useful data argument. A phone scan can seed more than a photorealistic demo: with geometry, feasibility filters, and an oracle, it can define a structured local training distribution. The strongest result comes when broad embodiment adaptation and scene personalisation are treated as separate datasets rather than collapsed into one fine-tuning folder. The remaining proof is operational—release the generator, failures, licences, and lineage needed to show that the reported pipeline can be reproduced outside the authors' campus.