DORA: Dexterous Offline Reinforcement Learning for Vision-Language-Action Models with Integrated Token-Action Critic
DORA improves dexterous VLA policies with token-grounded value learning, moving beyond imitation-only behavior and visually brittle external critics.
Abstract
While Vision-Language-Action (VLA) models excel in dexterous manipulation, successful task execution often exhibits multimodality, making reinforcement learning (RL) post-training essential for models to truly understand underlying action intents. However, applying RL is bottlenecked by Visual Value Hacking under severe visual occlusions, alongside credit assignment challenges in generative high-dimensional action spaces. To address this, we propose DORA, an offline RL framework for dexterous VLA models. It leverages a Consistency Policy to drastically shorten the action generation path and introduces a chunk-level Integrated Token-Action Critic. Real-robot experiments show DORA boosts the overall success rate from 40.8% to 60.0% over an imitation-only baseline, with a 25.0 percentage-point improvement on unseen objects and camera occlusion.
DORA Pipeline
Real-world Experiments
To complement the quantitative evaluation, we provide representative real-robot rollout sequences. Each row shows temporally ordered frames from one execution episode, covering successful task executions, object-unseen generalization, and a typical failure case.
(a) Pick and Place
A successful Pick and Place rollout, illustrating object approach, grasp establishment, transport, and placement.
(b) Pick Various Toys
A successful Pick Various Toys rollout, showing adaptation to irregular toy geometry.
(c) Flip Open the Box
A Flip Open the Box rollout, demonstrating articulated contact-rich manipulation under strong hand-object occlusion.
(d) Pick Various Toys (Object-Unseen)
A Pick Various Toys object-unseen rollout, illustrating evaluation on a toy object instance not observed during training.
These qualitative results demonstrate the policy's capability to dynamically adjust topological grasps in contact-rich real-world manipulation, while also revealing current physical limitations under extreme occlusion or under-actuated grasp maintenance.
Comparison
We compare the representation structure between an external critic and DORA. The performance gap directly substantiates the necessity of the Integrated Token-Action Critic: relying on raw image inputs and an independent encoder makes the external critic susceptible to visual value deception, while DORA assigns credit on a grounded physical semantic manifold by coupling chunk-level value estimation with VLM cognition tokens.
BibTeX
Coming Soon