Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis

Yafei Hu*1, Quanting Xie*1, Vidhi Jain *1,
Jonathan Francis 1,2, Jay Patrikar 1, Nikhil Keetha 1, Seungchan Kim 1, Yaqi Xie 1, Tianyi Zhang 1, Hao-Shu Fang 3, Shibo Zhao 1, Shayegan Omidshafiei4, Dong-Ki Kim4;
Ali-akbar Agha-mohammadi4, Katia Sycara1 Matthew Johnson-Roberson1 Dhruv Batra5,6 Xiaolong Wang7 Sebastian Scherer1 Chen Wang 8, Zsolt Kira5 Fei Xia9 † Yonatan Bisk1,6 †
1 Carnegie Mellon University , 2 Bosch Center for AI , 3 MIT , 4 Field AI , 5 Georgia Tech , 6 FAIR at Meta , 7 UC San Diego , 8 SAIR Lab , 9 Google DeepMind

*Equal contribution †Equal advising

Paper arXiv Paper List in the Survey

Abstract

Building general-purpose robots that can operate seamlessly, in any environment, with any object, and utilizing various skills to complete diverse tasks has been a long-standing goal in Artificial Intelligence. Unfortunately, however, most existing robotic systems have been constrained—having been designed for specific tasks, trained on specific datasets, and deployed within specific environments. These systems usually require extensively-labeled data, rely on task-specific models, have numerous generalization issues when deployed in real-world scenarios, and struggle to remain robust to distribution shifts. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i.e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of robotics, and also exploring (ii) what a robotics-specific foundation model would look like. We begin by providing an overview of what constitutes a conventional robotic system and the fundamental barriers to making it universally applicable. Next, we establish a taxonomy to discuss current work exploring ways to leverage existing foundation models for robotics and develop ones catered to robotics. Finally, we discuss key challenges and promising future directions in using foundation models for enabling general-purpose robotic systems. We encourage readers to view our living GitHub repository of resources, including papers reviewed in this survey as well as related projects and repositories for developing foundation models for robotics.

Challenges on General-purpose Robots


Taxonomy of the challenges in robotics that could be resolved by foundation models. We list five major challenges in the second level and some, but not all, of the keywords for each of these challenges.

Current Research

Taxonomy of General-purpose Robotics via Foundation Models

Conceptual Framework of Foundation Models in Robotics: The figure illustrates a structured taxonomy of foundational models, categorized into two primary segments: the application of existing foundation models (vision and language models) to robotics, and the development of robotic-specific foundation models. This includes distinctions between vision and language models used as perception tools, in planning, and in action, as well as the differentiation between single-purpose and general-purpose robot foundation models.

 

Brief Summary of Input and Output Complexity

Here we plot some representative works of foundation models used in robotics, and robotic foundation models. The horizontal axis represents the complexity of input data; the vertical axis represents the complexity of output action space. The complexity of the input data is brought by the modality of the data, e.g. image data is more complex than the text data. The complexity of output action space is mainly determined by the output dimension of the foundation model in the robotic tasks.

BibTeX

@article{hu2023robofm,
  author    = {Yafei Hu and Quanting Xie and Vidhi Jain and Jonathan Francis and Jay Patrikar and 
              Nikhil Keetha and Seungchan Kim and Yaqi Xie and Tianyi Zhang and Hao-Shu Fang and Shibo Zhao 
              and Shayegan Omidshafiei and Dong-Ki Kim and Ali-akbar Agha-mohammadi and Katia Sycara and 
              Matthew Johnson-Roberson and Dhruv Batra and Xiaolong Wang and Sebastian Scherer and Chen Wang 
              and Zsolt Kira and Fei Xia and Yonatan Bisk},
  title     = {Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis},
  booktitle = {arXiv preprint: arXiv:2312.08782 },
  year      = {2023},
}