ROBOCASA GR1 TABLETOP
84.7%
+5.9 percentage points
VAE-only: 78.8%
24 tasks · 50 trials per task
VISUAL REPRESENTATIONS FOR ROBOT LEARNING
What Visual Representations
Do World Action Models Need?
What to represent. How to integrate. When to combine.
1 Institute of Automation, Chinese Academy of Sciences2 Galbot3 Peking University4 University of Science and Technology of China5 Individual Researcher
arXiv:2610.13024 · October 2026
Motivation
The gap between
reconstruction and manipulation.
World action models inherit a VAE latent space optimized for visual reconstruction. Yet reconstructing what a scene looks like does not necessarily capture the information needed to act within it. So a natural question is: what visual representations do WAMs need?
We study this question within a common WAM framework, comparing representation sources, feature integration mechanisms, and combinations of sources. The result is a practical Representation Recipe — R²-WAM — validated across WAM formulations, simulation benchmarks, and physical robot tasks.
13pretrained visual encoders
5feature integration mechanisms
3research questions
View full figure THREE QUESTIONS, THREE FINDINGS
A closer look at
what makes a difference.
Explore the evidence behind the recipe.
Success rates (%) and gains in percentage points (pp).
RQ1 · WHICH REPRESENTATIONS HELP?
Start with a strong
general-purpose representation.
Across the evaluated encoders, general-purpose representations achieve a higher average LP success rate than specialized representations: 82.2% versus 80.0%. A general-purpose encoder has the highest observed mean in each of the seven perturbation categories.
+9.3pp
DINOv3 over the VAE-only baseline
84.0% vs. 74.7% overall success
Single-source comparison
Gain over VAE-only (pp) · channel-wise concatenation · means over three evaluation seeds
VAE-only baseline: 74.7%
Bars: gain over baseline · SR: success rateRQ2 · HOW SHOULD FEATURES BE INTEGRATED?
Predict future features.
Integrate them through shared visual tokens.
Channel-wise concatenation gives the highest average LP success across thirteen sources: 81.5%, or 5.1 percentage points above the next-best mechanism. The DINOv3 ablation supports future-feature prediction as a major contributor.
Current only76.5%
Future only81.9%
Current + future84.0%
The five integration mechanisms
View full figure ↗Performance, ablations, and action sensitivity
View full figure ↗RQ3 · IS ADDING MORE ALWAYS BETTER?
Complementarity matters.
More sources do not always help.
DINOv3 provides the main improvement over VAE-only: +9.3 percentage points. The best evaluated combination adds a further 3.7 points. The benefit of each additional source depends on the representations already present.
For example, adding SigLIP 2 changes success by +1.8 points with DINOv3, −0.4 with DINOv3 + UniTok, and +0.2 with DINOv3 + UniTok + V-JEPA 2.1.
View full figure ↗FROM FINDINGS TO A RECIPE
Introducing R²-WAM.
A compact combination of visual representations,
integrated through channel-wise concatenation.
VISUAL SELF-SUPERVISIONDINOv3
+MULTI-OBJECTIVE LEARNINGUniTok
+VIDEO PREDICTIVE LEARNINGV-JEPA 2.1
→CHANNEL-WISE CONCATENATIONR2-WAM
The three-source recipe reaches 87.3% on LIBERO-Plus. Adding HRP reaches 87.7%; we retain three sources to limit auxiliary encoders and feature-prediction targets.
VALIDATION BEYOND THE MAIN STUDY
The same recipe.
Different settings.
Improvements over corresponding VAE-only baselines
across simulation benchmarks and WAM formulations.
ROBOTWIN 2.0
92.8%
+2.2 percentage points
VAE-only: 90.6%
50 tasks · clean and randomized settings
OTHER WAM FORMULATIONS
Direct action prediction
70.8 → 79.4%
Video-then-action (IDM)
73.6 → 87.7%
LIBERO-Plus success · VAE-only → + recipe
ON A PHYSICAL ROBOT
From visual features
to everyday actions.
Four manipulation tasks on Galbot G1.
20 evaluation trials per task, per model.
Demonstrations from the paper’s supplementary material 01:34
| Task | Metric | VAE-only | R²-WAM |
|---|---|---|---|
| Air-fryer food preparation | Complete-task success | 50.0% | 60.0% |
| Instructed pick-and-place | Complete-task success | 70.0% | 80.0% |
| Tool sweeping | Mean debris collection rate | 48.3% | 66.7% |
| Cup pushing | Complete-task success | 65.0% | 75.0% |
Sweeping measures the proportion of six debris pieces collected per trial; the other tasks measure complete-task success.
REFERENCE THIS WORK
A recipe for
further research.
@misc{liu2026r2wam,
title = {What Visual Representations Do World Action Models Need?},
author = {Kai Liu and Jiangran Lyu and Tingyu Deng and Yixin Zheng
and Jingtian Wan and Wenbo Cui and Mi Yan and Yizhou Zhou
and Haoran Li and Dongbin Zhao and Zhizheng Zhang and He Wang},
year = {2026},
eprint = {2610.13024},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2610.13024}
}