Words demonstrate their value as robot-teaching aids.

The learning of a simulated robotic arm lifting and utilizing various tools can be sped up by employing human-language descriptions of the items, according to Princeton researchers who are investigating novel ways to train robots.

The findings add to the body of research showing that giving autonomous robots greater knowledge during artificial intelligence (AI) training can increase their ability to adapt to new circumstances and increase both their safety and effectiveness

Robot manipulation of newly encountered tools that were not in the first training set was enhanced by including descriptions of a tool’s form and function in the training process. At the Conference on Robot Learning on December 14, a group of mechanical engineers and computer scientists introduced the new approach, Accelerated Learning of Tool Manipulation with Language, or ATLA.

Robotic arms offer a tremendous deal of promise to assist with tedious or difficult activities, however teaching robots to use tools properly is challenging: Tools come in a broad variety of shapes, and a robot cannot match a human’s skill and perception.

According to research coauthor Anirudha Majumdar, an assistant professor of mechanical and aerospace engineering at Princeton who directs the Intelligent Robot Motion Lab, “additional information in the form of language can help a robot learn to use the tools more quickly.

The team used GPT-3, a sizable language model released by OpenAI in 2020 that makes use of deep learning to produce text in response to a prompt, to query for tool descriptions. They tried out a number of different prompts before settling on “Describe the [feature] of [tool] in a detailed and scientific response,” where the feature was the instrument’s design or intended use.

According to Karthik Narasimhan, assistant professor of computer science and study coauthor, “because these language models were trained on the internet, in some ways you can think of this as a different way of retrieving that information,” more effectively and comprehensively than using crowdsourcing or scraping particular websites for tool descriptions. As a visiting research scientist at OpenAI, Narasimhan—a lead faculty member in Princeton’s natural language processing (NLP) group—contributed to the development of the initial GPT language model.

The research teams of Narasimhan and Majumdar have never worked together before. Majumdar’s research focuses on creating AI-based policies to assist robots, such as walking and flying robots, generalize their capabilities to novel environments. He was interested in how recent “huge gains in natural language processing” can aid robot learning.

The group decided to use a training set of 27 items, ranging from an axe to a squeegee, for their tests on simulated robot learning. They assigned the robotic arm four distinct tasks: pushing, lifting, sweeping a cylinder over a table, and driving a peg into a hole. In order to examine how well the policies performed on a different test set of nine tools with paired descriptions, the researchers built a set of rules using machine learning training methodologies with and without language input.

Since the robot becomes better at learning with each new task, this method is referred regarded as meta-learning. It is not only learning how to use each tool, but Narasimhan added that it is also “trying to learn to understand the descriptions of each of these hundred distinct tools, so that when it sees the 101st tool it will learn to use the new tool faster.” We’re teaching the robot how to utilize the tools and also teaching it English at the same time.

With the nine test tools, the researchers evaluated the robot’s performance in pushing, lifting, sweeping, and hammering. They contrasted the outcomes obtained with machine learning policies that employed linguistic information to those that did not. The majority of the time, the language knowledge provided the robot with considerable advantages when using new tools.

Allen Z. Ren, a Ph.D. student in Majumdar’s lab and the paper’s primary author, noted that using a crowbar to sweep a cylinder, or bottle, around a table, was one job that demonstrated noticeable differences between the policies.

Ren said that through linguistic training, the crowbar learns to grip at the long end and use the curved surface to better restrain the movement of the bottle. Without language, it held the crowbar tightly against the curved surface and was more difficult to handle.

The study, which is a component of a larger TRI-sponsored initiative in Majumdar’s research group aimed at enhancing robots’ capacity to operate in novel circumstances that are unlike from their training environments, was financed in part by the Toyota Research Institute (TRI).

The overarching objective, according to Majumdar, is to enable robotic systems, notably those that have undergone machine learning training, to generalize to new settings. His team’s other TRI-funded research has focused on failure prediction for vision-based robot control and employed a “adversarial environment generation” strategy to make robot policies more effective in situations that aren’t familiar to them.

Leveraging language for rapid tool manipulation learning was the title of the article that was presented on December 14 at the Conference on Robot Learning. Bharat Govil, a member of the Princeton Class of 2022, and Tsung-Yen Yang, who finished his doctoral work in electrical engineering at Princeton this year and is currently a machine learning scientist at Meta Platforms Inc., are the other coauthors in addition to Majumdar, Narasimhan, and Ren.

The Office of Naval Research, the U.S. National Science Foundation, and the School of Engineering and Applied Science at Princeton University, thanks to the generosity of William Addy ’82, provided funding for the study in addition to TRI.