Scaling Laws for Multilingual Code Pretraining

This episode explores whether code models should allocate training data evenly across programming languages or tune the mix based on how each language scales. It walks through the paper’s core experiments on monolingual scaling, bilingual transfer, translation-style code pairing, and multilingual token allocation, with close attention to languages like Python, JavaScript, Rust, and TypeScript. The discussion highlights the paper’s main argument that programming languages differ in scaling behavior and cross-lingual usefulness, which could make non-uniform token budgets more effective than treating all code equally. Listeners would find it interesting for its concrete take on how multilingual software ecosystems challenge simple scaling-law assumptions and for its careful scrutiny of whether the evidence really supports those stronger optimization claims. Sources: 1. Scaling Laws for Multilingual Code Pretraining https://arxiv.org/pdf/2512.13472 2. Unsupervised Translation of Programming Languages — Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, Guillaume Lample, 2020 https://scholar.google.com/scholar?q=Unsupervised+Translation+of+Programming+Languages 3. XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence — Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, Chandan K. Reddy, 2022 https://scholar.google.com/scholar?q=XLCoST%3A+A+Benchmark+Dataset+for+Cross-lingual+Code+Intelligence 4. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation — Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, Abhinav Jangda, 2022 https://scholar.google.com/scholar?q=MultiPL-E%3A+A+Scalable+and+Extensible+Approach+to+Benchmarking+Neural+Code+Generation 5. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X — Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, Jie Tang, 2023 https://scholar.google.com/scholar?q=CodeGeeX%3A+A+Pre-Trained+Model+for+Code+Generation+with+Multilingual+Benchmarking+on+HumanEval-X 6. Scaling Laws for Code: A More Data-Hungry Regime — Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan, Wanxiang Che, 2025 https://scholar.google.com/scholar?q=Scaling+Laws+for+Code%3A+A+More+Data-Hungry+Regime 7. Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study — Razan Baltaji, Saurabh Pujar, Louis Mandel, Martin Hirzel, Luca Buratti, Lav Varshney, 2025 https://scholar.google.com/scholar?q=Cross-lingual+Transfer+in+Programming+Languages%3A+An+Extensive+Empirical+Study 8. CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution — Ruiyang Xu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun, 2024 https://scholar.google.com/scholar?q=CRUXEval-X%3A+A+Benchmark+for+Multilingual+Code+Reasoning%2C+Understanding+and+Execution 9. RepoTransBench: A Real-World Benchmark for Repository-Level Code Translation — Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, Zibin Zheng, 2024 https://scholar.google.com/scholar?q=RepoTransBench%3A+A+Real-World+Benchmark+for+Repository-Level+Code+Translation 10. Towards Multi-Language Repository-Level Code Generation: From-Scratch to Guided Tasks — authors unclear from snippet, unknown https://scholar.google.com/scholar?q=Towards+Multi-Language+Repository-Level+Code+Generation%3A+From-Scratch+to+Guided+Tasks 11. Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and Beyond — authors unclear from snippet, unknown https://scholar.google.com/scholar?q=Do+Not+Treat+Code+as+Natural+Language%3A+Implications+for+Repository-Level+Code+Generation+and+Beyond 12. Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models — authors unclear from snippet, unknown https://scholar.google.com/scholar?q=Code-Switching+In-Context+Learning+for+Cross-Lingual+Transfer+of+Large+Language+Models 13. Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer — authors unclear from snippet, unknown https://scholar.google.com/scholar?q=Bridging+the+Language+Gap%3A+Enhancing+Multilingual+Prompt-Based+Code+Generation+in+LLMs+via+Zero-Shot+Cross-Lingual+Transfer 14. Advancing Code Translation With Context-Aware Pre-Training in Data-Scarce Environments — authors unclear from snippet, unknown https://scholar.google.com/scholar?q=Advancing+Code+Translation+With+Context-Aware+Pre-Training+in+Data-Scarce+Environments 15. Semi-Supervised Code Translation Overcoming the Scarcity of Parallel Code Data — authors unclear from snippet, unknown https://scholar.google.com/scholar?q=Semi-Supervised+Code+Translation+Overcoming+the+Scarcity+of+Parallel+Code+Data 16. Scaling Laws for Predicting Downstream Performance in LLMs — authors unclear from snippet, unknown https://scholar.google.com/scholar?q=Scaling+Laws+for+Predicting+Downstream+Performance+in+LLMs 17. AI Post Transformers: CODEGEN: Open Language Model for Code Synthesis — Hal Turing & Dr. Ada Shannon, Fri, https://podcast.do-not-panic.com/episodes/codegen-open-language-model-for-code-synthesis/ 18. AI Post Transformers: Llama 3: Architecture, Capabilities, and Safety — Hal Turing & Dr. Ada Shannon, Sun, https://podcast.do-not-panic.com/episodes/llama-3-architecture-capabilities-and-safety/ 19. AI Post Transformers: Program Synthesis with Large Language Models — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-20-program-synthesis-with-large-language-mo-b962ec.mp3 20. AI Post Transformers: MTEB & MMTEB: The Massive Text Embedding Benchmark — Hal Turing & Dr. Ada Shannon, Fri, https://podcast.do-not-panic.com/episodes/mteb-mmteb-the-massive-text-embedding-benchmark/ Interactive Visualization: Scaling Laws for Multilingual Code Pretraining