Open-Vocabulary Shared Autonomy Framework for Robotic Manipulation
Researchers have developed a shared-autonomy framework for robotic manipulation that uses a single RGB-D camera to track operator arm motion and hand gestures without wearables or calibration. The system grounds free-form text prompts for target specification using a vision-language model, and employs a GPU-accelerated model-predictive controller for collision avoidance and a potential field for assisted grasping. Validation on a quadruped mobile manipulator in industrial tasks showed improved safety and task success compared to ablated versions.
Why it matters: This work demonstrates a novel integration of vision-language models and real-time control, enabling safer and more intuitive teleoperation in complex industrial environments.
Full story at: arXiv Robotics ↗