SFMP Search-Free Mixed-Precision LLM Quantization

This episode explores SFMP, a post-training, weight-only quantization method for large language models that targets awkward memory budgets like 3.25 or 3.75 bits per weight, where standard 3-bit or 4-bit schemes are a poor fit. It explains the core ideas behind mixed-precision quantization, including salience scoring with diagonal Fisher information, block-wise allocation, and why hardware constraints make many fine-grained methods difficult to deploy in practice. The discussion centers on SFMP’s main claim: instead of running an expensive search over bit allocations, it ranks blocks by importance and assigns only the two neighboring integer precisions, such as giving the most salient 25 percent of blocks 4-bit weights and the rest 3-bit for a 3.25 BPW target. Listeners would find it interesting because it frames quantization not as a narrow accuracy trick, but as a concrete engineering problem about trading off model quality, memory limits, and real serving-system compatibility. Sources: 1. SFMP Search-Free Mixed-Precision LLM Quantization https://arxiv.org/pdf/2602.01027 2. HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision — Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, Kurt Keutzer, 2019 https://scholar.google.com/scholar?q=HAWQ%3A+Hessian+AWare+Quantization+of+Neural+Networks+with+Mixed-Precision 3. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2023 https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers 4. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, 2024 https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+LLM+Compression+and+Acceleration 5. AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models — Sangjun Lee, Seung-taek Woo, Jungyu Jin, Changhun Lee, Eunhyeok Park, 2025 https://scholar.google.com/scholar?q=AMQ%3A+Enabling+AutoML+for+Mixed-precision+Weight-Only+Quantization+of+Large+Language+Models 6. BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments — Xinghao Wang, Pengyu Wang, Bo Wang, Dong Zhang, Yunhua Zhou, Xipeng Qiu, 2024 https://scholar.google.com/scholar?q=BitStack%3A+Fine-Grained+Size+Control+for+Compressed+Large+Language+Models+in+Variable+Memory+Environments 7. SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models — Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, Xiaojuan Qi, 2024 https://scholar.google.com/scholar?q=SliM-LLM%3A+Salience-Driven+Mixed-Precision+Quantization+for+Large+Language+Models 8. FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs — Xilong Xie, Liang Wang, Limin Xiao, Meng Han, Lin Sun, Shuai Zheng, Xiangrong Xu, 2025 https://scholar.google.com/scholar?q=FineQ%3A+Software-Hardware+Co-Design+for+Low-Bit+Fine-Grained+Mixed-Precision+Quantization+of+LLMs 9. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2024 https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge 10. MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design — Zhen Zheng et al., 2024 https://arxiv.org/abs/2412.14590 11. SpinQuant: LLM quantization with learned rotations — Zechun Liu et al., 2024 https://arxiv.org/abs/2405.16406 12. DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization — Yuantian Shao et al., 2025 https://arxiv.org/abs/2511.04063 13. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — Coleman Hooper et al., 2024 https://arxiv.org/abs/2401.18079 14. QAQ: Quality Adaptive Quantization for LLM KV Cache — Shichen Dong et al., 2024 https://arxiv.org/abs/2403.04643 15. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3 16. AI Post Transformers: PackKV Lossy Compression for KV Caches — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-04-packkv-lossy-compression-for-kv-caches-b37bce.mp3 17. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3 18. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3