Architectural Darwinism or Computational Profligacy

Send us Fan MailThis podcast, takes a look at the application of Neural Architecture Search as a form of evolutionary algorithm or brute-force search to improve Large Language Model architectures, The idea that random modification of these architectures could result in long term improvements is compelling in its simplicity, however the case for it is flawed due to the astronomical costs of training, and the Neural Scaling Laws identifying that the primary driver of LLM performance is scale and not their architectures.