Neural Architecture Search
Lilian Weng
Neural Architecture Search (NAS) is a machine learning approach that automatically discovers optimal neural network architectures rather than relying on human expert design. The field has three major components: search space (defining valid operations and connections), search algorithm (sampling and optimizing architecture candidates), and evaluation strategy (measuring performance). Different NAS approaches use varied search space representations—including sequential layer-wise operations, cell-based modules, hierarchical structures, and memory-bank models—combined with search algorithms like random search and reinforcement learning, with applications ranging from CNNs to RNNs that can transfer across datasets.
Why it matters
Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.
