EP054: Alpaca - Stanford Built a $600 GPT Clone

Alpaca is a 7-billion-parameter instruction-following language model developed by researchers at Stanford CRFM. It was created by fine-tuning Meta's LLaMA 7B model on 52,000 instruction-following demonstrations. These demonstrations were automatically generated using OpenAI's text-davinci-003 via a simplified "self-instruct" method, which cost less than $500 to produce. The fine-tuning process took only about 3 hours and cost less than $100, making the entire model surprisingly cheap (under $600) and easy to reproduce.Despite its small size and low training budget, preliminary blind human evaluations revealed that Alpaca performs qualitatively similarly to OpenAI's text-davinci-003, winning 90 versus 89 comparisons.The primary goal of the Alpaca project is to provide the academic community with an accessible, high-performing model to study the capabilities and flaws of instruction-following AI, a space previously dominated by closed-source models. The creators emphasize that Alpaca is strictly for academic research and non-commercial use. Like other large language models, Alpaca exhibits several known limitations, including hallucination, toxicity, social stereotypes, and the ability to generate well-written misinformation. By releasing the training recipe, data generation code, and 52K data demonstrations, the authors hope to empower researchers to perform controlled scientific studies to better understand and mitigate these pressing safety issues.