All Episodes

Qwen3.8-Next

Distillation Scaling Laws

Decode-Branch Transformers

The State-Prediction Separation Hypothesis

Pretraining Mixture of Experts for Emergent Modularity

Muon-Optimized Distillation and Quantization for LLM Deployment

Continuous-Query Limited Memory Language Models

QuaRot

How to set AdamW’s weight decay as you scale model and dataset size

Weight Decay Improves Language Model Plasticity

Bridging Compute- and Data-Optimal Pretraining

Pushing LLM Pretraining Scales

Why Larger Models Learn More

Rethinking the Role of Efficient Attention in Hybrid Architectures

Nemotron-Labs-Diffusion

Lightning Index Ultra-Long Context via Lookahead Sparse Attention

Learning beyond Teacher

Value Residual Learning

Learning Rates Regulate Catastrophic Overtraining

HRM-Text

Why Warmup the Learning Rate

Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

SonicMoE- Accelerating MoE with IO and Tile-aware Optimizations

Memory Sparse Attention Model

Shaping capabilities with token-level data filtering

Self-Improving Pretraining

Scale Dependent Data Duplication

Rare Tokens Degenerate All Tokens

Perplexity Cannot Always Tell Right from Wrong

Neural Neural Scaling Laws

Mamba 3

M2RNN

Lost in Backpropagation- The LM Head is a Gradient Bottleneck

Let's (not) just put things in Context- Test-Time Training for Long-Context LLMs

Learning State-Tracking from Code Using Linear RNNs

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining

GLM-5

Cautious Optimizers

Backward Gradient Normalization in Deep Neural Networks

Attention Residuals

Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings

Scaling Laws for Precision

On the "Induction Bias" in Sequence Models

NOBLE- Accelerating Transformers with Nonlinear Low-Rank Branches

Flash Attention 4

An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence

Midtraining Bridges Pretraining and Posttraining Distributions

SiameseNorm

ÜberWeb

Why Do Reasoning Models Loop

OPUS- Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

Teon

Cautious Weight Decay

Predictable Scale

A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs

Challenges and Research Directions for Large Language Model Inference Hardware

Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

The Quantization Model of Neural Scaling

EAGLE-3

Engram Paper

From Entropy to Epiplexity- Rethinking Information for Computationally Bounded Intelligence

Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration

NorMuon- Making Muon more efficient and scalable

Dion- Distributed Orthonormalized Updates

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining-v5

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining-v4

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining-v3

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining-v2

Key and Value Weights Are Probably All You Need

Latent State Models of Training Dynamics

DeepSeek OCR

The Coverage Principle - How Pre-training Enables Post-Training

The Coverage Principle- How Pre-training Enables Post-Training

The Art of Scaling Reinforcement Learning Compute for LLMs

Continual Learning via Sparse Memory Finetuning

DeepSeek OCR Paper

Untitled Episode

Characterization and Mitigation of Training Instabilities in Microscaling Formats

Demystifying Synthetic Data in LLM Pre-training- A Systematic Study of Scaling Laws, Benefits, and Pitfalls

Drop-Muon- Update Less, Converge Faster

Data Mixing Can Induce Phase Transitions in Knowledge Acquisition

Apertus Tech Report

Learning Facts at Scale with Active Reading

Fantastic Pretraining Optimizers and Where to Find Them

Benchmarking Optimizers for Large Language Model Pretraining

Learning Facts at Scale with Active Reading.old

Apertus Tech Report.old

Benchmarking Optimizers for Large Language Model Pretraining.old

Fantastic Pretraining Optimizers and Where to Find Them.old

The Pitfalls of Next-Token Prediction

Large Language Models and Games

UQ - Assessing Language Models on Unsolved Questions

UQ- Assessing Language Models on Unsolved Questions

Signal and Noise - A Framework for Reducing Uncertainty in Language Model Evaluation

Signal and Noise- A Framework for Reducing Uncertainty in Language Model Evaluation

Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training

Thinking Like Transformers

Kimi K2

ERNIE Technical Report

Dynamic Chunking for End-to-End Hierarchical Sequence Modeling