Plenty of companies have demoed home robots, but few, if any, have actually been released. The truth is that even the most advanced robots aren't very good at interacting with objects and environments designed for humans. Researchers at MIT are trying to change that. A team from MIT's Electrical Engineering and Computer Science (EECS) is using large language models (LLMs) to give robots the "common sense knowledge" they need to be helpful around the house.
Robots can be programmed to complete any physical task that is within their physical limitations, and they'll perform it perfectly every time as long as nothing changes. Robots aren't good at coping with the unexpected, which is why Boston Dynamics has gotten so much attention for designing robots that don't fall over. That's not a high bar, but it's still rare because these problems are so difficult to solve.
The MIT engineers working on this project have found a way to connect a robot's physical motion with AI models that have thus far been used to generate content. This approach allows the robot to split a task into subtasks. It takes things one step at a time and can adjust to unexpected events without starting over. Importantly, the designers don't have to program robots with fixes for every possible eventuality.
Most of the LLMs we've seen use their data libraries to map the connections between words, allowing them to generate new words—or images, or computer code, or whatever they've been trained to make. The MIT study, led by graduate student Yanwei Wang, replaced words with subtasks. The system was tested with a robotic arm attempting to scoop marbles from one bowl to another. The LLM might produce a sequence like "reach" or "pour," which is mapped to the robot's physical movements.
The robot's algorithm is known as a "grounding classifier," meaning it can learn how to identify subtasks based on where the arm is in space. The team started by guiding the arm through the scooping task and then used a pre-trained LLM to list subtasks. The algorithm was able to match the subtasks to the robot's physical movements. With that done, the researchers allowed the robot to go about its business, scooping marbles and dumping them in another bowl. When the robot had that figured out, the team began interrupting the arm and nudging it out of alignment. Traditional control algorithms would have needed to go back to a known starting point, but the LLM-powered robot was able to understand where it was during each disruption and could simply pick up where it left off.
Wang notes this work could lead to household helpers that can adapt to their environment and better cope with external complications. Humans don't need to program for every eventuality if a robot can learn with the help of an AI model.
