Qwen3.8-Next
from Mechanical Dreams Hosted by Mechanical Dirk
In this episode: • Welcome and the 1/9th FLOP Miracle: Linda introduces Qwen3.8-Next and its massive efficiency gains, hooking Dr. Norris. • Token Mixing with GDN and Sparse Attention: Discussing the Gated DeltaNet hybrid architecture and how Qwen Sparse Attention scales to long contexts. • The Gated Residual (GR): Norris is skeptical about widening the residual stream, but Linda explains the gating mechanism and how it boosts stability. • N-gram Embeddings & The Muon Optimizer: Exploring off-accelerator capacity scaling and why Muon needs special engineering like the Canzona framework. • Hyperparameters & Conclusion: The removal of batch-size warmup, the importance of joint optimization, and final takeaways.
Embed this episode
No transcript yet.