Do Large Language Model Benchmarks Test Reliability?

The paper highlights the lack of reliable benchmarks for large language models, proposing "platinum benchmarks" to minimize label errors and revealing persistent model failures in simple tasks. https://arxiv.org/abs//2502.03461 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers