Eigenvectors of Experts: Training-free MoE Routing Without Collapse

Talent identification collapses in Sparse Mixture-of-Experts models when different experts' outputs drift toward near-identical functions, quietly wasting the parameter capacity that makes MoE architectures like Mixtral, DeepSeek-MoE, and GPT-OSS efficient. This episode covers "Eigenvectors of Experts are Training-free Non-collapsing Routers," which finds this collapse present across ten current frontier MoE models spanning a few billion to over 120 billion parameters, including GPT-OSS-120B, the Qwen3-MoE family, and ERNIE-4.5. The discussion explains why collapse is more than an efficiency loss — it erodes the interpretability guarantees needed in regulated domains like medicine and law, where practitioners want to trace a decision to a specific specialized expert. The paper's proposed fix skips retraining entirely, instead reading routing decisions directly off the eigenvectors already latent in each expert's trained weight matrix, on the reasoning that specialization acquired during training is already encoded in which input directions an expert's weights respond to most strongly. The conversation walks through the mechanics of Sparse Mixture-of-Experts and conditional computation before unpacking why a training-free, geometry-based router is both a practical and theoretically grounded departure from prior collapse fixes like HyperRouter and StableMoE. Sources: 1. Eigenvectors of Experts are Training-free Non-collapsing Routers — Giang Do, Hung Le, Truyen Tran, 2026 http://arxiv.org/abs/2605.30992 2. Your mixture-of-experts LLM is secretly an embedding model for free — Li, Z. and Zhou, T., 2025 https://scholar.google.com/scholar?q=Your+mixture-of-experts+LLM+is+secretly+an+embedding+model+for+free 3. On the representation collapse of sparse mixture of experts — Chi, Z. et al. (XMoE), 2022 https://scholar.google.com/scholar?q=On+the+representation+collapse+of+sparse+mixture+of+experts 4. Dropping experts, recombining neurons: Retraining-free pruning for sparse mixture-of-experts LLMs — Zhou, Y. et al., 2025 https://scholar.google.com/scholar?q=Dropping+experts%2C+recombining+neurons%3A+Retraining-free+pruning+for+sparse+mixture-of-experts+LLMs 5. Small singular values matter: A random matrix analysis of transformer models — Staats, M., Thamm, M., and Rosenow, B., 2026 https://scholar.google.com/scholar?q=Small+singular+values+matter%3A+A+random+matrix+analysis+of+transformer+models 6. SVD-LLM v2: Optimizing singular value truncation for large language model compression — Wang, X., Alam, S., Wan, Z., Shen, H., and Zhang, M., 2025 https://scholar.google.com/scholar?q=SVD-LLM+v2%3A+Optimizing+singular+value+truncation+for+large+language+model+compression 7. Zero-shot sparse mixture of low-rank experts construction from pre-trained foundation models (SMILE) — Tang, A. et al., 2026 https://scholar.google.com/scholar?q=Zero-shot+sparse+mixture+of+low-rank+experts+construction+from+pre-trained+foundation+models+%28SMILE%29 8. Tight clusters make specialized experts — Nielsen, S., Teo, R., Abdullaev, L., and Nguyen, T. M., 2025 https://scholar.google.com/scholar?q=Tight+clusters+make+specialized+experts 9. Accuracy is not all you need — Dutta, A., Krishnan, S., Kwatra, N., and Ramjee, R., 2024 https://scholar.google.com/scholar?q=Accuracy+is+not+all+you+need Interactive Visualization: Eigenvectors of Experts: Training-free MoE Routing Without Collapse