Julia, the robot above, stirs a mixing bowl in A.V. Williams.
The lab looked like the set of a cooking show from the future.
At the center was Julia, standing behind a table stocked with utensils and ingredients. An audience gathered around in a semicircle to see her prepare her next dish.
Julia might seem like a celebrity chef, but she has hulking scarlet arms and is made of metal and plastic.
Julia is an industrial robot.
“I am looking for the tomatoes,” Julia, a Baxter robot built by Rethink Robotics, said in her Siri-esque voice, just like a very thorough Cooking Channel host who announces even the smallest details to the audience.
Julia, named after the famous chef Julia Child, then grasped a plastic container of tomatoes, lifted it over the salad bowl and dumped in the contents. For this demonstration, Julia made a fairly minimalist salad — tomatoes stirred in the bowl — and she only mimed pouring the dressing so the bowl wouldn’t need cleaning afterward.
But Julia can also prepare coffee and serve drinks, with more recipes to come in the future. And she has learned all her culinary skills by watching videos on YouTube.
A team of researchers from the Autonomy Robotics Cognition Lab within the University of Maryland Institute for Advanced Computer Studies has written a paper about how they engineered Julia and another identical robot to learn cooking techniques and recipes after watching them online.
Much as humans might visit YouTube for guitar lessons or makeup tips, these robots can take what they see in YouTube videos and apply them to their own actions. With image-recognition software, the objects and actions in the video are translated into a grammar of commands the robots can follow, said Yezhou Yang, a computer science doctoral candidate and the lead author of the paper.
From the video, the robot knows which action – such as grasp, pour or pound – to complete and what object to complete it on.
“For example, if you want to ask robot to stir a salad, the robot needs to know the first action and grasp type to use on the ladle, and then uses another grasp type to hold the bowl,” he said. “We want to learn these kinds of schemas from online videos.”
Using a database of 88 cooking videos, the researchers established a source from which the robots could potentially one day learn to cook a whole repertoire of recipes, said Cornelia Fermüller, an associate research scientist with UMIACS and co-author of the paper.
Robots learning from human behavior is nothing new, said S.K. Gupta, the Maryland Robotics Center director. The field is known as imitation learning, or learning from demonstration, and it could play a significant role in the future of robotics.
“So imagine if you get a robot at home,” Gupta said, “and if you want the robot to learn a new skill, it’s much easier for you to show the robot how to do the task than it is to program it.”
Kinesiology professor Rodolphe Gentili has also worked with imitation learning in the past, but he uses it as a tool to understand the human brain. He said researchers could learn a lot if robots could completely replicate human motor action.
“We may not necessarily reach that,” he said. “The idea is that this will provide a step that will hopefully allow us to better understand the brain as well as to hopefully develop better control systems.”
While many imitation learning projects focus on mimicking direct action, this UMIACS project is more related to goal-oriented work, Gupta said. A robot might not necessarily be able to copy the exact action of a human it watches, but it could figure out how to work toward the same goal.
Fermüller said this also allows the robot to adapt to less-than-ideal cooking scenarios.
For instance, the robot can learn to use a spatula instead of a knife if a knife is not available, Yang said.
“The robot can also reason,” Fermüller said. “So if a tomato disappears, if someone ate it, what does it do now?”
It may be some time before humans are served meals by animatronic Gordon Ramsays, but Yang noted this was one step further toward robots that are able to complete more complex actions. This could lead to an advancement of artificial intelligence, or skills like pouring drinks could contribute to another kind of progress.
“This could be the first step to having a robotic bartender,” Yang said.
