Muon-Optimized Distillation and Quantization for LLM Deployment

In this episode: • Introduction & The Edge Deployment Problem: Professor Norris and Linda introduce the episode's paper, focusing on the difficulty of running large language models on edge devices with limited memory and compute. • The Multi-Teacher Distillation Pipeline: Linda explains the dual-teacher approach, using Llama 4 Scout for synthetic data generation and Llama 3.3 70B for logit-based knowledge distillation to a 3B parameter student. • Hyperparameter Surprises with Optuna: The hosts discuss the surprising finding that Bayesian optimization consistently drops the cross-entropy loss in favor of pure Kullback-Leibler divergence from the teacher. • The Muon Optimizer and Quantization Robustness: Linda introduces the Muon optimizer and explains how fine-tuning with it dramatically reduces accuracy decay when the model is compressed to 4-bit weights using GPTQ. • Results & Conclusion: A summary of the final throughput and latency gains, proving that highly specialized, edge-ready LLMs can achieve double the speed and half the memory without sacrificing accuracy.