Quality-Diversity Optimisation on a Physical Robot Through Dynamics-Aware and Reset-Free Learning

Smith, Simón C.; Lim, Bryan; Janmohamed, Hannah; Cully, Antoine

Abstract

Learning algorithms, like Quality-Diversity (QD), can be used to acquire repertoires of diverse robotics skills. This learning is commonly done via computer simulation due to the large number of evaluations required. However, training in a virtual environment generates a gap between simulation and reality. Here, we build upon the Reset-Free QD (RF-QD) algorithm to learn controllers directly on a physical robot. This method uses a dynamics model, learned from interactions between the robot and the environment, to predict the robot's behaviour and improve sample efficiency. A behaviour selection policy filters out uninteresting or unsafe policies predicted by the model. RF-QD also includes a recovery policy that returns the robot to a safe zone when it has walked outside of it, allowing continuous learning. We demonstrate that our method enables a physical quadruped robot to learn a repertoire of behaviours in two hours without human supervision. We successfully test the solution repertoire using a maze navigation task. Finally, we compare our approach to the MAP-Elites algorithm. We show that dynamics awareness and a recovery policy are required for training on a physical robot for optimal archive generation. Video available at https://youtu.be/BgGNvIsRh7Q

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: Genetic and Evolutionary Computation Conference Companion (GECCO 2023 Companion)
• Start Date: Jul 15, 2023
• End Date: Jul 19, 2023
• Acceptance Date: Jun 1, 2023
• Online Publication Date: Jul 24, 2023
• Publication Date: 2023
• Deposit Date: Jul 11, 2023
• Publicly Available Date: Jul 24, 2023
• Publisher: Association for Computing Machinery (ACM)
• Pages: 171-174
• Book Title: GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation
• ISBN: 9798400701207
• DOI: https://doi.org/10.1145/3583133.3590625
• Keywords: Quality-Diversity, Robotics, Real-time optimisation