245: Benchmarking DNA foundation models

Feng H et al., Nat Commun - A comprehensive, unbiased benchmark compares five DNA foundation models across 57 datasets and multiple tasks, finding mean token embeddings improve classification and that model strengths vary by task and pre-training. Key terms: DNA foundation models, mean token embedding, sequence classification, variant effect, gene expression. Study Highlights: The study evaluated DNABERT-2, NT-v2, HyenaDNA, Caduceus-Ph, and GROVER on 57 datasets spanning sequence classification, gene expression prediction, variant effect quantification, and TAD recognition. Mean token embedding consistently and significantly outperformed summary-token and max pooling for sequence classification. Model performance was task-dependent: Caduceus-Ph excelled at human TFBS and promoter tasks, NT-v2 led pathogenic variant identification, HyenaDNA scaled efficiently and benefited from multi-species pre-training, while specialized models outperformed general foundations on QTL prediction. Zero-shot embeddings provided modest gene expression prediction and NT-v2 attention patterns did not reveal inherent TAD recognition. Conclusion: Mean token pooling yields more robust sequence-level representations and model choice should align with task, input length, and pre-training data for best genomic performance Music: Enjoy the music based on this article at the end of the episode. First author: Feng H Journal: Nat Commun DOI: 10.1038/s41467-025-65823-8 Reference: Feng H, Wu L, Zhao B, Huff C, Zhang J, Wu J, Lin L, Wei P & Wu C. Benchmarking DNA foundation models for genomic and genetic tasks. Nat Commun. 2025;16:10780. https://doi.org/10.1038/s41467-025-65823-8 License: This episode is based on an open-access article published under the Creative Commons Attribution 4.0 International License (CC BY 4.0) – https://creativecommons.org/licenses/by/4.0/ Support: Base by Base – Stripe donations: https://donate.stripe.com/7sY4gz71B2sN3RWac5gEg00 Official website https://basebybase.com On PaperCast Base by Base you’ll discover the latest in genomics, functional genomics, structural genomics, and proteomics. Episode link: https://basebybase.com/episodes/dna-foundation-models-benchmark QC: This episode was checked against the original article PDF and publication metadata for the episode release published on 2025-12-31. QC Scope: - article metadata and core scientific claims from the narration - excludes analogies, intro/outro, and music - transcript coverage: Audited the transcript's coverage of core scientific claims: DNA foundation models benchmarking, pooling strategies (mean token embedding), zero-shot embeddings with a downstream classifier, VEQ dichotomy, multispecies pre-training and cross-species generalization, long-sequence performance, TAD recognition limitations - transcript topics: DNA foundation models and zero-shot embeddings; Pooling strategies for sequence representations (mean token embedding vs summary/max pooling); Downstream classification using zero-shot embeddings (random forest); Variant effect quantification: pathogenic vs QTL (VEQ dichotomy); Multispecies pre-training and cross-species generalization; Cross-species transfer in promoter identification (Arabidopsis example) QC Summary: - factual score: 10/10 - metadata score: 10/10 - supported core claims: 8 - claims flagged for review: 0 - metadata checks passed: 4 - metadata issues found: 0 Metadata Audited: - article_doi - article_title - article_journal - license Factual Items Audited: - Mean token embedding consistently improves sequence classification across all foundation models and yields measurable AUROC gains. - Zero-shot embeddings with frozen weights are evaluated with a downstream random forest classif...