Channel Configuration for Neural Architecture: Insights from the Search Space

Thomson, Sarah L.; Ochoa, Gabriela; Veerapen, Nadarajen; Michalak, Krzysztof

Abstract

We consider search spaces associated with neural network channel configuration. Architectures and their accuracy are visualised using low-dimensional Euclidean embedding (LDEE). Optimisation dynamics are captured using local optima networks (LONs). LONs are a compression of a fitness landscape: the nodes are local optima and the edges are search transitions between them. Several neural architecture search algorithms are tested on the search space and we discover that iterated local search (ILS) is a competitive algorithm for neural channel configuration. We additionally implement a landscape-aware ILS which performs well. Observations from the search and landscape space analyses bring visual clarity and insight to the science of neural network channel design: the results indicate that a high number of channels, kept constant throughout the network, is beneficial.

Citation

Thomson, S. L., Ochoa, G., Veerapen, N., & Michalak, K. (2023, July). Channel Configuration for Neural Architecture: Insights from the Search Space. Presented at GECCO '23, Lisbon, Portugal

• Presentation Conference Type: Conference Paper (published)
• Conference Name: GECCO '23
• Start Date: Jul 15, 2023
• End Date: Jul 19, 2023
• Acceptance Date: Apr 1, 2023
• Online Publication Date: Jul 12, 2023
• Publication Date: 2023-07
• Deposit Date: Aug 16, 2023
• Publicly Available Date: Aug 17, 2023
• Publisher: Association for Computing Machinery (ACM)
• Pages: 1267-1275
• Book Title: GECCO '23: Proceedings of the Genetic and Evolutionary Computation Conference
• ISBN: 9798400701207
• DOI: https://doi.org/10.1145/3583131.3590386
• Keywords: Fitness Landscapes, Neural Architecture Search, Local Optima Networks (LONs)