#5 - Humanity's Last Exam: A Multimodal Benchmark
from Artificially Speaking Hosted by Henry Moran
Humanity's Last Exam (HLE) is a new, extremely difficult, multi-modal benchmark designed to evaluate large language models (LLMs). Created by a global team of experts, HLE features 3,000 questions spanning numerous academic subjects, surpassing the capabilities of current LLMs. The benchmark's questions were rigorously tested and reviewed to ensure difficulty and eliminate easy solutions, such as internet searches. HLE's public release aims to provide a standardized measure of LLM capabilities for researchers and policymakers, while a private test set guards against overfitting. The low accuracy and high calibration errors demonstrated by state-of-the-art LLMs on HLE highlight a significant gap between current AI capabilities and human expertise.
Embed this episode
No transcript yet.