EP074: How Gemini Beat Human Experts

The paper introduces Gemini, a new family of natively multimodal models developed by Google, designed to seamlessly understand and reason across text, image, audio, and video.The Gemini 1.0 family is built to accommodate different computational limitations and is released in three sizes: Ultra for highly complex reasoning tasks, Pro for efficient deployability at scale, and Nano for on-device applications.Key highlights of the paper include:State-of-the-Art Performance: Gemini Ultra advances the state-of-the-art in 30 out of 32 prominent benchmarks across text, reasoning, image, video, and speech. Notably, it is the first model to achieve human-expert performance on the MMLU (Massive Multitask Language Understanding) exam benchmark, scoring over 90%.Native Multimodality: Unlike previous models that stitch together separate vision and language components, Gemini is jointly trained across modalities from the very beginning. This allows it to exhibit impressive cross-modal reasoning, natively ingesting interleaved sequences of audio, images, and text, and directly outputting images using discrete image tokens.Complex Reasoning and Coding: The models show strong proficiencies in mathematics, science, and coding. When combined with search and tool-use mechanisms, Gemini powers advanced agents like AlphaCode 2, which performs in the top 15% of entrants in competitive programming.Safety and Responsible Deployment: After large-scale pre-training, the models undergo rigorous post-training—utilizing Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF)—to improve overall quality, ensure alignment, and mitigate potential harms.The models are deployed through two main variants: Gemini Apps models (optimized for conversational AI services like Gemini Advanced) and Gemini API models (optimized for developers building applications via Google AI Studio and Cloud Vertex AI).