Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

This research introduces the concept of Neural Thickets, describing a phenomenon where large pretrained models are surrounded by a high density of diverse, task-specific solutions in their local weight space. While small models require structured optimization like gradient descent to find improvements, larger models transition into a regime where random weight perturbations frequently yield "expert" versions of the model. The authors exploit this discovery through RandOpt, a parallel post-training method that samples random weight changes, selects the best performers, and ensembles their predictions. Their findings show that these random experts are specialists rather than generalists, often excelling at one task while declining in others, which makes ensembling via majority vote highly effective. This approach proves competitive with standard reinforcement learning methods like PPO and GRPO, especially as model scale increases. Ultimately, the study suggests that sufficient pretraining fundamentally reshapes the loss landscape, making complex downstream adaptation possible through simple parallel search and selection.