#11 - Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

This research paper investigates the effectiveness of ensembling different large language models (LLMs) to improve performance. The authors introduce Self-MoA, a method that aggregates multiple outputs from a single, top-performing LLM, contrasting it with traditional Mixture-of-Agents (MoA) which combines outputs from multiple LLMs. Experiments across various benchmarks show Self-MoA significantly outperforms MoA in many cases, highlighting the importance of model quality over diversity. A sequential version of Self-MoA is also presented to address scalability issues. The study explores the trade-off between diversity and quality in ensemble methods, ultimately demonstrating Self-MoA's superior performance and efficiency.