EP094: DeepSeek-V3 Rivals GPT-4 for $6 Million

The "DeepSeek-V3 Technical Report" presents DeepSeek-V3, a highly efficient and powerful Mixture-of-Experts (MoE) language model with 671 billion total parameters, of which 37 billion are activated for each token.Key Highlights of DeepSeek-V3:Innovative Architecture: The model retains the Multi-head Latent Attention (MLA) and DeepSeekMoE architectures validated in DeepSeek-V2 for efficient inference and cost-effective training. Furthermore, it pioneers an auxiliary-loss-free load balancing strategy to prevent the performance degradation typically caused by forced load balancing, and it incorporates a Multi-Token Prediction (MTP) objective to enhance overall benchmark performance.Highly Efficient Training: DeepSeek-V3 was pre-trained on 14.8 trillion diverse tokens. By utilizing an FP8 mixed precision training framework and algorithmic innovations like the DualPipe algorithm for computation-communication overlap, the researchers achieved near-zero communication overhead across nodes. This resulted in an incredibly economical full training cost of only 2.788 million H800 GPU hours (approximately $5.576 million), with the process being remarkably stable and free of irrecoverable loss spikes.Advanced Post-Training: The post-training phase involved Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). A major post-training innovation was the knowledge distillation from the DeepSeek-R1 reasoning model, which elegantly incorporated reflection and verification patterns to significantly boost DeepSeek-V3's reasoning and coding capabilities.State-of-the-Art Performance: Comprehensive evaluations reveal that DeepSeek-V3 is the strongest open-source base model currently available. It comprehensively outperforms other open-source models and achieves performance comparable to leading closed-source models—such as GPT-4o and Claude-3.5-Sonnet—across a wide array of educational, math, code, and factual knowledge benchmarks.