Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving
Researchers introduce a fast-slow architecture for autonomous driving that separates a frozen 7B vision-language model (operating at 5 Hz) from a lightweight action expert (operating at 20 Hz). The slow model provides a cached scene representation, while the fast expert uses this cache and the current camera frame to generate waypoints at every simulation tick. In CARLA simulations on LangAuto-Short routes, this approach achieves 94% route completion compared to 37% for a frame-skipping baseline, and per-tick freshness reduces red-light violations by a third. The system also demonstrates strong zero-shot transfer to unseen towns and significantly lowers waypoint error compared to the backbone model's own action head.
Why it matters: This work offers a practical solution to bridging the latency gap between large language model inference and real-time control in autonomous driving, enabling substantial improvements in closed-loop performance and generalization.
Full story at: arXiv Robotics ↗