Frontiers in synthetic data
from Interconnects Hosted by Nathan Lambert
Synthetic data is known to be a super powerful tool for every level of the language modeling stack. It's documented as being used for expanding vanilla pretraining data and creating large swaths of fine-tuning data. Many, many more rumors surround its use, Anthropic's pretraining-scale constitutional AI, Mistral AI's first models being pretrained on OpenAI outputs, Q-star's hopes as OpenAI's remaining moat, and much more. The diversity of use cases for synthetic data makes planning around the role of synthetic data in solving specific goals. This is AI generated audio with Python and 11Labs. Source code: https://github.com/natolambert/interconnects-tools Original post: https://www.interconnects.ai/p/frontiers-in-synthetic-data 00:00 Frontiers in synthetic data 01:14 1. Direct distillation is still king 02:54 2. Are Gemini Flash and Claude Haiku distilled? 04:03 3. Filtering prevents collapse 06:30 4. Synthetic data strategy taxes 07:32 5. Pros and cons of training on multi-output-source synthetic datasets 08:54 6. Structured synthetic data 09:42 7. Weak-to-strong generalization is maybe real 10:27 8. Creating synthetic prompts is overlooked again This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.interconnects.ai/subscribe
Embed this episode
No transcript yet.