EP076: OLMo Cracks Open the AI Black Box

The paper introduces OLMo, a state-of-the-art, truly open language model designed to accelerate the scientific study of large language models. While the commercial value of language models has led to the most powerful models being closed off or only partially released (e.g., releasing only weights or inference code), OLMo provides the research community with full access to its entire development framework.Key highlights of the paper include:A Fully Open Framework: The release encompasses the complete pipeline, including the model weights (in 1B and 7B variants), the open pretraining dataset called Dolma, training and evaluation code, detailed training logs, and hundreds of intermediate model checkpoints. All code and weights are released under a permissive Apache 2.0 license.Competitive Performance: The OLMo-7B model, trained on at least 2 trillion tokens, is highly competitive with other similarly sized models like LLaMA-7B, Llama-2-7B, MPT-7B, and Falcon-7B across various zero-shot downstream tasks and perplexity benchmarks.Adaptation and Alignment: The authors also fine-tuned OLMo using instruction tuning (SFT) and Direct Preference Optimization (DPO). The adapted models showed significant improvements in general chat capabilities, safety, and truthfulness, proving that OLMo serves as a very strong base model for downstream applications.Research Goal: By releasing everything from the exact datasets used to the intermediate training steps, the creators of OLMo aim to help researchers study poorly understood aspects of language models, such as how training data impacts model capabilities, the effects of hyperparameter choices, and the models' underlying biases and risks.