Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis

Yafei Hu*1, Quanting Xie*1, Vidhi Jain *1,
Jonathan Francis 1,2, Jay Patrikar 1, Nikhil Keetha 1, Seungchan Kim 1, Yaqi Xie 1, Tianyi Zhang 1, Shibo Zhao 1, Yu Quan Chong 1, Chen Wang 3, Katia Sycara1 Matthew Johnson-Roberson1 Dhruv Batra4,5 Xiaolong Wang6 Sebastian Scherer1 Zsolt Kira4 Fei Xia7 † Yonatan Bisk1,5 †
1Carnegie Mellon University, 2Bosch Center for AI, 3SAIR Lab 4Georgia Tech
5FAIR at Meta 6UC San Diego 7Google DeepMind

*Equal contribution †Equal advising

Paper arXiv Full Paper List

Abstract

Building general-purpose robots that can operate seamlessly, in any environment, with any object, and utilizing various skills to complete diverse tasks has been a long-standing goal in Artificial Intelligence. Unfortunately, however, most existing robotic systems have been constrained—having been designed for specific tasks, trained on specific datasets, and deployed within specific environments. These systems usually require extensively-labeled data, rely on task-specific models, have numerous generalization issues when deployed in real-world scenarios, and struggle to remain robust to distribution shifts. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i.e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of robotics, and also exploring (ii) what a robotics-specific foundation model would look like. We begin by providing an overview of what constitutes a conventional robotic system and the fundamental barriers to making it universally applicable. Next, we establish a taxonomy to discuss current work exploring ways to leverage existing foundation models for robotics and develop ones catered to robotics. Finally, we discuss key challenges and promising future directions in using foundation models for enabling general-purpose robotic systems. We encourage readers to view our living GitHub repository of resources, including papers reviewed in this survey as well as related projects and repositories for developing foundation models for robotics.

BibTeX

@article{hu2023Toward,
  author    = {Yafei Hu and Quanting Xie and Vidhi Jain and Jonathan Francis and Jay Patrikar 
                and Nikhil Keetha and Seungchan Kim and Yaqi Xie and Tianyi Zhang and Shibo Zhao 
                and Yu-Quan Chong and Chen Wang and Katia Sycara and Matthew Johnson-Roberson 
                and Dhruv Batra and Xiaolong Wang and Sebastian Scherer and Zsolt Kira and 
                Fei Xia and Yonatan Bisk},
  title     = {Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis},
  booktitle = {arXiv preprint: arXiv:2312.08782 },
  year      = {2023},
}