Critical Points of Random Neural Networks

This work examines the critical points of random neural networks, particularly as network depth increases in the infinite-width limit. The authors provide asymptotic formulas for the expected number of critical points, categorized by their index or when exceeding a threshold. Their analysis reveals three distinct scaling regimes for the expected number of critical points based on a specific derivative of the covariance function. Theoretical findings are corroborated by numerical experiments, which also suggest potential divergence in critical points for networks with irregular activation functions like ReLU.