Data & collection
Language annotation
A language annotation is natural-language metadata attached to a robot-data sample, segment, or episode. It may state the instruction given before execution, describe what happened afterward, name a task or subtask, identify objects, record a correction, or explain an outcome. These annotation types are not interchangeable because they contain different information and may be available at different times.
Also known as: natural-language annotation, task-language annotation, language label, language annotations
Updated
Instructions and descriptions carry different information
An instruction records what the robot was asked to do before or during the attempt. A retrospective description records what an annotator believes happened after seeing the result. A failed episode can therefore have the instruction “put the cup in the sink” while an accurate description says the robot dropped it beside the sink.
Task names, object labels, subtask boundaries, corrections and outcome explanations add other forms of supervision. A dataset should store them as separate fields rather than flattening every string into one “language” column.
Annotation timing and authorship matter
BridgeData V2 added natural-language labels after collection through crowdsourcing, asking annotators to describe the performed task. DROID provides several natural-language descriptions for a trajectory. These labels support language-conditioned learning, but they do not necessarily reproduce the operator's exact original intent.
AgiBot World 2026 exposes task-level and finer instruction segments with frame boundaries. Segment-level language can support hierarchical policies only when those boundaries align with the underlying action sequence.
Quality depends on grounding, not fluency
Useful language annotations name the relevant action, objects, relations and success state consistently. They should preserve uncertainty and failure rather than rewrite every episode as a successful demonstration.
Documentation should identify who or what produced each label, when it was written, the annotation prompt or schema, review process, vocabulary policy and temporal scope. Automatically generated captions should remain distinguishable from human instructions and verified labels.
Sources
Related terms
Data & collection
Robot training data
Robot training data is recorded experience used to train, fine-tune, or adapt models for robot perception, prediction, planning, or control. It can include sensor observations, robot state, actions, task instructions, rewards or outcomes, demonstrations, failures, and embodiment metadata. Not every dataset contains every field, but their timing and physical meaning must be clear.
Data & collection
Demonstration
A demonstration is a recorded example of how an intended task or behaviour is performed, usually represented as a time-aligned sequence of observations, states and actions. For humanoid robot learning, demonstrations may come from teleoperation, kinaesthetic guidance, motion capture or autonomous experts and provide targets for imitation.
Models & learning
Vision-language-action model
A vision-language-action model is a robot policy that conditions on visual observations and natural-language instructions to predict executable actions. It adapts a pretrained vision-language model, or a similar multimodal backbone, using robot trajectories so that semantic knowledge is connected to an embodiment’s action space. The action representation may use discrete tokens or continuous values.
Data & collection
Trajectory
A trajectory is a time-ordered sequence of states or observations, actions and, where applicable, rewards generated as an agent or robot evolves. A complete episode or policy rollout often yields a trajectory, but the terms are not universally identical: trajectories may be partial, while episodes have dataset- or environment-defined boundaries.
Models & learning
Long-horizon task
A long-horizon task is a temporally extended robot task whose success depends on maintaining reliable behaviour across many actions, phases, or dependent subtasks. The term has no universal step-count threshold: it usually signals sequential dependencies, accumulating execution error, delayed outcomes, changing object state, or information that must be remembered beyond the current observation.