A Comprehensive Evaluation of Quantized Instruction-Tuned LLMs

📏 A Comprehensive Evaluation of Quantized Instruction-Tuned LLMsThis paper, titled "A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B," examines the performance of large language models (LLMs) after they have been compressed using various quantization methods. The authors assess the impact of these techniques on different task types and model sizes, including the very large 405B parameter Llama 3.1 model. They explore how different quantization methods, model sizes, and bit-widths affect performance, finding that larger quantized models often outperform smaller FP16 models and that certain methods, such as weight-only quantization, are particularly effective for larger models. The study also concludes that task difficulty does not significantly impact the accuracy degradation caused by quantization.📎 Link to paper