Sample and time efficient policy learning with CMA-ES and Bayesian Optimisation

Le Goff, Léni K.; Buchanan, Edgar; Hart, Emma; Eiben, Agoston E.; Li, Wei; De Carlo, Matteo; Hale, Matthew F.; Angus, Mike; Woolley, Robert; Timmis, Jon; Winfield, Alan; Tyrrell, Andrew M.

Abstract

In evolutionary robot systems where morphologies and controllers of real robots are simultaneously evolved, it is clear that there is likely to be requirements to refine the inherited controller of a 'newborn' robot in order to better align it to its newly generated morphology. This can be accomplished via a learning mechanism applied to each individual robot: for practical reasons, such a mechanism should be both sample and time-efficient. In this paper, We investigate two ways to improve the sample and time efficiency of the well-known learner CMA-ES on navigation tasks. The first approach combines CMA-ES with Novelty Search, and includes an adaptive restart mechanism with increasing population size. The second bootstraps CMA-ES using Bayesian Optimisation, known for its sample efficiency. Results using two robots built with the ARE project's modules and four environments show that novelty reduces the number of samples needed to converge, as does the custom restart mechanism; the latter also has better sample and time efficiency than the hybridised Bayesian/Evolutionary method.

Citation

Le Goff, L. K., Buchanan, E., Hart, E., Eiben, A. E., Li, W., De Carlo, M., Hale, M. F., Angus, M., Woolley, R., Timmis, J., Winfield, A., & Tyrrell, A. M. (2020, July). Sample and time efficient policy learning with CMA-ES and Bayesian Optimisation. Presented at ALife 2020, Online

• Presentation Conference Type: Conference Paper (published)
• Conference Name: ALife 2020
• Start Date: Jul 13, 2020
• End Date: Jul 18, 2020
• Acceptance Date: Jun 1, 2020
• Online Publication Date: Jul 14, 2020
• Publication Date: 2020-07
• Deposit Date: Jul 15, 2020
• Publicly Available Date: Jul 15, 2020
• Publisher: MIT Press
• Pages: 432-440
• Book Title: ALIFE 2020: The 2020 Conference on Artificial Life
• DOI: https://doi.org/10.1162/isal_a_00299
• Public URL: http://researchrepository.napier.ac.uk/Output/2675888
• Publisher URL: https://www.mitpressjournals.org/toc/isal/32