Testing Synthetic Data Generators Before You Scale

Picking a synthetic data generator by model reputation is a trap — a brilliant problem-solver can produce terrible training material. This episode digs into AgoraBench, a benchmark that trains the same student model on each generator's output and measures how much of the performance gap it closes. We cover why zero-seed generation is a different problem from transforming real data, how creating new instances and refining existing ones are separate skills, and why diversity often collapses as you scale from 2,000 to 20,000 samples. Plus: what a small pilot actually needs to prove before you commit to a big batch. Key sources: • AgoraBench — https://arxiv.org/abs/2412.03679 • A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data (LLM Data Auditor), arXiv:2601.17717, TMLR 06/2026 — https://arxiv.org/abs/2601.17717 • Invent a Dataset: Measuring dataset generation abilities with zero seed data, arXiv:2610.01674 — https://arxiv.org/html/2610.01674 Episode 6239 — myweirdprompts.com/n/6239 (permanent short link: myweirdprompts.com/533936)