All episodes

Gemini 2.5

How new data permeates LLM knowledge and how to dilute it

Harnessing the Universal Geometry of Embeddings

Model Merging in Pre-training of Large Language Models

Learning Dynamics in Continual Pre-Training for Large Language Models

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs

Scalable-Softmax Is Superior for Attention

Breast Cancer Recurrence Prediction

Shareable artificial intelligence to extract cancer outcomes from electronic health records for precision oncology research

Native Sparse Attention

Critical Batch Size Revisited

Base of RoPE Bounds Context Length

Rope to Nope and Back Again

Training Deep Learning Models with Norm-Constrained LMOs

SkyLadder

LLMs on the Line

The Leaderboard Illusion

Why Linearly Decaying the Learning Rate to Zero Works Best

Not All Data Are Unlearned Equally

A Multi-Power Law for Loss Curve Prediction

Efficient Training of Ultra-Long Context Large Language Models

Multi-Token Attention

From Style to Facts

Compute Optimal Scaling of Skills

Predictive Data Selection

Continual Pre-training of MoEs

s1 - Simple test-time scaling

Cognitive Behaviors that Enable Self-Improving Reasoners

Phi 4 Multimodal Instruct

Claude 3.7 Sonnet System Card

Project Sid: Many-agent simulations toward AI civilization

The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training

Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

NExtLong - Toward Effective Long-Context Training without Long Documents

Optimizing Pretraining Data Mixtures with LLM-Estimated Utility

Over-Tokenized Transformer

HashAttention: Semantic Sparsity for Faster Inference

From Tokens to Words

DeepSeek V3

Optimal Linear Decay Learning Rate Schedules and Further Refinements

Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers

Efficient and Approximate Per-Example Gradient Norms for Gradient Noise Scale

Phi-4

Rephrasing natural text data with different languages and quality levels

Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs

EXAONE 3.5

Model soups - averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Model-Aware Data Selection for Efficient Pretraining with Data Influence Models

Nemotron-CC

Tülu 3

The Zamba2 Suite

Small-scale proxies for large-scale Transformer training instabilities

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

Understanding WSD Learning Rates

Toward Understanding Why Adam Converges Faster Than SGD for Transformers

Amuro & Char - Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models

Evaluation data contamination in LLMs: How do we measure it and (when) does it matter?

How Does Critical Batch Size Scale in Pre-training?

The Road Less Scheduled

Learning-Rate-Free Learning by D-Adaptation

Scaling FP8 Training to Trillion Token LLMs

Geometric Dynamics of Signal Propagation Predict Trainability of Transformers

A Survey on Model MoErging

Liquid Time-constant Networks

Scaling Laws for Predicting Downstream Performance in LLMs

A Spectral Condition for Feature Learning

Don't decay the learning rate

OLMoE

An Empirical Model of Large Batch Training