Facebook has been working on advancing embodied AI, which focuses on training versatile robots that will be able to navigate in real-world spaces alongside humans.
onurdongel / Getty Images
Facebook has announced a new step in the company’s journey towards achieving what it calls “embodied AI” – the technology that could one day enable robots to carry out tasks in mundane settings, such as stocking the fridge with groceries or taking out the trash.
The social media platform unveiled Habitat 2.0, an upgrade on the Habitat simulation platform that lets researchers train their robots at speed in virtual environments, which render with utmost precision the smallest details that the machines are likely to be faced with in an ordinary setting like a kitchen or a living room.
Letting the robots roll about freely in Habitat’s simulated virtual environments comes with evident cost and time savings compared to training them in a real-world setting, and Facebook is hopeful that the platform will fast-track the development of assistive robots that can help out with household chores.
For many years, Facebook has been working on advancing embodied AI, which focuses on training versatile robots that will be able to navigate in real-world spaces alongside humans, reacting to previously unknown environments or unexpected objects coming in their way.
This would mean, for example, that house robots could help their human fetch objects on command or load the dishwasher, but could also open the door to even more disruptive technologies like a guide robot to assist the visually impaired when walking outdoors.
But to help the machines accomplish useful real-world tasks, it is necessary to provide them with experience across hundreds of different real-world environments – complete with baby toys scattered on the floor and upturned carpet corners.
This is where simulation can be a game-changer. Rather than bringing a physical robot to different flats, houses and offices over the course of months and years, scientists at Facebook believe that a much more pragmatic approach consists of exposing the AI to virtual environments to speed up its training.
Two years ago, therefore, the social media giant unveiled the first version of Habitat, which included the Replica dataset – a compilation of 18 3D scans of scenes ranging from office conference rooms to two-storey houses.
Replica was described as an ultra-realistic library of settings, capturing some of the most subtle details in each scene such as mirror reflections and rug textures. With a significant caveat: Replica was a static dataset. In other words, while the robot could navigate around the virtual space, it could not interact with any of the objects.
With Habitat 2.0, Facebook has now started to tackle this challenge. Not only can robots roll around the new platform’s virtual environments, but they can also interact with objects that they would find in a normal kitchen, dining room or other commonly used space.
“With this new data set and platform, AI researchers can go beyond just building virtual agents in static 3D environments and move closer to creating robots that can easily and reliably perform useful tasks like stocking the fridge, loading the dishwasher, or fetching objects on command and returning them to their usual place,” said Dhruv Batra, research scientist at Facebook.
The upgrade required refreshing Replica, which now comes under the name of ReplicaCAD and supports the movement and manipulation of objects.
ReplicaCAD is the result of over 900 hours of work by 3D artists, who created 111 unique layouts of living space including 92 objects. Specific attention was given to the material composition, geometry and texture of the objects, as well as whether the items have specific mechanisms, such as opening and closing for doors and refrigerators.
The artists went so far as re-creating realistic clutter such as kitchen utensils, books and furniture, said Facebook.
The platform is still far from perfect: for example, ReplicaCAD doesn’t support non-rigid dynamics like liquids, cloths or ropes; nor does it include audio and tactile sensing. Still, Habitat 2.0 enables robots to virtually carry out a variety of tasks, ranging from setting the table, cleaning the fridge, cleaning the house or running other common household errands.
Facebook is not the only company taking interest in virtual simulations for embodied AI. Earlier this year, for example, the Seattle-based Allen Institute for AI released ManipulaTHOR, a sophisticated virtual robot arm capable of manipulating objects in more than 100 simulated environments.
Where the social media company may score extra points, however, is in the speed of Habitat 2.0, which Batra said is two orders of magnitude faster than most 3D simulators currently available to researchers.
For example, the platform can simulate a Fetch robot interacting in ReplicaCAD at 1,200 steps per second (SPS), while other platforms run at ten to 400 SPS. With the right combination of GPUs, the platform can even reach up to 26,000 SPS.
“Such speeds significantly cut down on experimentation time, allowing researchers to complete experiments that would typically run over 6 months in as little as 2 days,” said Batra. “Reduced experiment times will allow researchers to try new ideas much more early and often, leading to the completion of a far greater number of simulations and paving the way for more advances in the field.”
Facebook’s research team still has a long way to go before Habitat can reach its maximum capabilities. Speed, of course, can always be improved, and Batra referred to specific bottlenecks such as long load times when an episode resets.
Another key challenge will be to link different tasks together. While robots can currently be trained for individual skills like picking, placing or opening a drawer, they are yet to be taught how to accomplish many of these tasks and chain them without accumulating errors, which still significantly limits the technology’s usefulness.
Facebook is also keen to expand ReplicaCAD and to incorporate more living spaces that reflect different cultures, layouts and types of objects. “We acknowledge these representational challenges and are working to improve the diversity and geographic inclusion of the 3D environments currently available for research,” said Batra.
To that end, the company has opened Habitat to third-party 3D assets, and announced a partnership with spatial data company Matterport. The deal will see Matterport making a collection of 1,000 digital twins generated from real-world environments available to the platform, encompassing bedrooms, bathrooms, kitchens and hallways of various styles, sizes and complexities.
