VLA-Δ

Multilevel Physical State Transition Learning

VLA-Δ: Learning Physical State Transition for Vision-Language-Action Models

Don't just imitate actions. Understand change.

Qi LuJiarun LiCe ZhangMiao PanYi PanXu WangWenqi ZhangXuhong Zhang
Zhejiang University (zju.edu.cn)
GitHub Hugging Face Project Page arXiv

01 · Introduction

Why physical state transitions matter

Conceptual animation · not experimental footage.Download MP4 ↓

VLAs may close the gripper before reaching the target object, or continue toward the placement location after a failed grasp. These failures suggest that a policy may replay a memorized trajectory rather than respond to the evolving scene. Most VLAs lack explicit supervision of intermediate physical state changes. VLA-Δ uses these changes as explicit learning targets, combining task semantics and visual motion to learn how the robot and its environment should change as the task progresses.

Comparison of transition modeling paradigms: Direct VLA, Reasoning VLA, World Action Models (WAMs), Latent-WAMs, and VLA-Delta with multilevel physical state-transition supervision.

  1. Physical state-transition learning

    Models task semantics and visual motion changes for continuous action generation.

  2. Automated data annotation pipeline

    Multilevel physical state-transition annotations from ≈484 hours and 24 million frames of robot demonstrations.

  3. Simulation and real-robot experiments

    Outperforms compared methods on LIBERO and RoboTwin, and π0.5 in real-robot experiments.

02 · Method

Multilevel Physical State Transition Learning

Physical State Difference Queries organize vision-language information into a physical state-transition representation. The action expert combines this representation with the independently encoded robot proprioceptive state to predict a continuous action chunk via flow matching.

Overall architecture of VLA-Delta: multi-view observations and a task instruction feed the Qwen3-VL backbone with Physical State Difference Queries. Structured Transition Slots, Optical Flow and Future-State Difference Text supervise the transition representation; the action expert also receives robot state. The right panel shows attention-visibility constraints.
Overall architecture of VLA-Δ. The model uses Qwen3-VL-2B-Instruct and a 9-layer Diffusion Transformer. Direct prediction reads designated Query hidden states to predict Structured Transition Slots and dense Optical Flow. Future-State Difference Text positions attend only to the Queries and preceding text.View full-size ↗
Predefined state variables

Structured Transition Slots

Nine categories of task-relevant state variables and their changes provide direct supervision for learning physically meaningful transition representations.

Dense visual motion

Optical Flow

Dense two-dimensional displacements between current and subsequent observations from the same viewpoint encourage Queries to preserve local motion details.

Coherent task semantics

Future-State Difference Text

Coherent natural language describes task progress, spatial changes, contact transitions and associated constraints, relating local state changes to overall task progression.

Physical state-transition annotations

Six annotated states for putting the apple into the container: Approach at 0.0 s, Approach at 2.5 s, Align at 3.8 s, Manipulate at 4.9 s, Release at 6.1 s, and Retract at 7.7 s. Each state shows the original progress, phase, gripper state and contact annotations.
View full-size ↗

Future-State Difference Text

Task: load the bowl

Current observation: a robot gripper holds the rim of a pink bowl near the tabletop, with a wire rack to the right.
Current state
Next observation: the closed gripper holds the pink bowl higher above the tabletop, near the wire rack.
Next state

Task_temporal The overall task is "load the bowl". At the current time, the task is in the "Carry the bowl toward the loading area." phase, with the total task progress at 0.42. The next subgoal is "Align the bowl with the loading area." The next total task progress is 0.50. Spatial_motion The target object is the pink bowl. At the current time, the pink bowl is visible in the center horizontal and lower-middle vertical band, with normalized bounding box [0.388, 0.550, 0.505, 0.685]. The gripper is to the right and above of the pink bowl, with normalized bounding box [0.402, 0.415, 0.615, 0.633]. At the next time, the pink bowl is visible in the center horizontal and middle vertical band, with normalized bounding box [0.527, 0.463, 0.645, 0.602]. The gripper is to the right and above of the pink bowl, with normalized bounding box [0.519, 0.157, 0.822, 0.427]. Contact_interaction During the next action steps, the gripper remains closed; the gripper moves right 8 cm. The gripper maintain contact with the pink bowl in this phase. The gripper maintains a closed grasp on the bowl's rim, lifting it from the table surface and moving it laterally toward the wire rack structure. The task-relevant affordance is the bowl_rim_grasp_region [0.458, 0.615] [0.492, 0.615] of the pink bowl. The detected object-state change for the pink bowl is from resting on table surface to suspended in mid-air during this phase. Object_constraints The gripper must maintain a secure closed grip on the bowl rim to prevent dropping during the transport and alignment maneuver. The bowl must remain stable within the gripper's grasp and not tilt excessively while being moved toward the target area.

Automated Data Annotation Pipeline

DROID, MolmoAct and RH20T demonstrations are standardized in LeRobot v3, annotated for current–future state pairs, and assessed through automated checks, model review and manual audits.

≈484hours of demonstrations
≈24Mframes
Automated Data Annotation PipelineEight steps: Data selection; Unified format; Data cleaning; Keyframes + semantics; Visual annotation; Sample construction; LLM review; Human review. LLM review feeds back to keyframes and semantics to revise prompts and rules. Human review audits 300 pairs, 100 each from DROID, MolmoAct and RH20T, across four text aspects and nine slot categories.1Data selectionDROIDMolmoActRH20TValid, diverse segments2Unified formatLeRobot v3Parquet + videosTimestamp alignment3Data cleaningLanguage Timing VisionState / action IntegritySplit trajectories before windows4Keyframes + semanticsQwen3.8-27BEvent-adaptive samplingObjects, phases, contact, constraints5Visual annotationSAM 3UniDepth V2MegaFlowBox + maskSurface depth2D flowShared object IDs and time6Sample constructiontt + ΔPrediction horizonFuture-State Difference Text{ }Structured Transition Slots7LLM reviewGPTField-by-field reviewReannotate, mask, or discard8Human review300pairsDROIDMolmoActRH20T1001001004 text aspects + 9 slot categoriesRevise prompts and rules
Eight annotation steps, with review feedback to revise prompts and rules.View full-size ↗

03 · Experiments

Simulation and Real-Robot Evaluation

98.4%LIBERO40 tasks · 2,000 episodes
90.06%RoboTwin · Clean50 tasks · 100 episodes/task
89.74%RoboTwin · Random50 tasks · 100 episodes/task
68.0%Real robot5 tasks · 200 trials

LIBERO: Spatial, Object, Goal and Long

The four suites assess spatial understanding, object manipulation, goal-conditioned task completion and long-horizon execution. VLA-Δ achieves an average success rate of 98.4%, the highest among all compared methods.

Performance comparison on LIBERO · Success Rates (%) · selected methods from Table 1
MethodsVLM SizeSpatialObjectGoalLongAverage
π0.53B98.898.298.092.496.9
GR00T-N1.72B97.797.598.594.497.0
openVLA-OFT7B97.698.497.994.697.1
VLA-Adapter0.5B97.899.297.295.097.3
ZR-02B97.499.497.896.497.8
VLA-Δ2B98.899.898.696.498.4

RoboTwin 2.0: Clean and Random

Clean uses relatively fixed scene appearances and object layouts. Random varies object instances, initial positions, lighting, backgrounds, tabletop layouts and instruction phrasing.

Performance comparison on RoboTwin 2.0 · average success rates (%) · all methods from Table 2
MethodCleanRandom
X-VLA72.8872.84
π065.9258.4
π0.582.7476.76
Motus88.5287.02
ZR-088.7087.98
VLA-Δ90.0689.74

LIBERO-Plus: generalization under environmental changes

89.0% under the Language perturbation, outperforming π0.5 at 82.6% by 6.4 percentage points.

Across the remaining perturbation settings, VLA-Δ remains competitive with leading methods.

50 episodes/task/perturbation · 8,000 episodes total.

Original LIBERO-Plus chart. VLA-Delta success rates are 89.0% for Language, 95.1% for Light, 94.8% for Background and 83.2% for Layout, with baseline comparisons.
Original paper Figure 4 · success rate (%).View full-size ↗
Ablation Experiments on LIBERO

The complete VLA-Δ model combines all three supervision types, achieving an average success rate of 98.40%. Future-State Difference Text, Structured Transition Slots and Dense Optical Flow each improve over Difference Queries Only with action-only pretraining (96.45%). All variants use only the action-learning objective during downstream fine-tuning.

Ablation results on 40 LIBERO tasks · complete Table 3
VariantQueryPretrainingDifference TextSlotsOptical FlowLIBERO Avg. (%)
Dense VLM FeaturesNoNoNoNoNo95.70
Dense VLM FeaturesNoYesNoNoNo95.85
Difference Queries OnlyYesNoNoNoNo96.25
Difference Queries OnlyYesYesNoNoNo96.45
Query + Difference TextYesYesYesNoNo97.20
Query + Structured SlotsYesYesNoYesNo96.90
Query + Optical FlowYesYesNoNoYes97.35
VLA-Δ (Full)YesYesYesYesYes98.40

04 · Experiments

Real-Robot Evaluation

Five manipulation tasks evaluate obstacle avoidance, long-horizon execution, fine-grained manipulation, spatial reasoning and generalization under environmental changes.

Experimental Results — real-robot success rates Success Rate (%), from 0 to 100. Four methods across five tasks and their average. Hover, focus or tap a bar to show its value. Each task has 40 evaluation episodes. (b) Experimental Results 0 25 50 75 100 Success Rate (%) Obstacle Avoidance Long-Horizon Execution Fine-Grained Manipulation Spatial Reasoning Generalization AVG
Paper Figure 5(b) · success rate (%) · 40 episodes per task.View full-size ↗

Real-robot demonstrations

Videos loop automatically. Hover to enlarge and pause scrolling; swipe to browse.

Obstacle Avoidance and Physical Constraint Compliance

Place the carrots on a rack, making sure they do not touch the cups.

Long-Horizon Execution

Put all the items on the tabletop onto the tray.

Fine-Grained Manipulation

Put the spoon into the cup.

Spatial Reasoning

Place the item closest to the bowl onto the plate.

Generalization under Changed Tablecloth

Place the item closest to the bowl onto the plate with a changed tablecloth.

05 · Citation

Cite this work

Please use the following citation when referring to this work.

BibTeX citation
@misc{lu2026vladeltalearningphysicalstate,
  title = {VLA-$\Delta$: Learning Physical State Transition for Vision-Language-Action Models},
  author = {Qi Lu and Jiarun Li and Ce Zhang and Miao Pan and Yi Pan and Xu Wang and Wenqi Zhang and Xuhong Zhang},
  year = {2026},
  eprint = {2610.21306},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2610.21306}
}