EPISODE · Aug 20, 2025 · 13 MIN
Hugging Face Hub Storage: Xet vs. Git LFS
from Neural intel Pod · host Neuralintel.org
The provided text discusses Hugging Face's storage solutions for large, binary files, specifically focusing on the transition from Git LFS to their new Xet-backed storage system. It explains how repositories handle massive files like models and datasets through pointer files that reference data stored remotely, typically on Amazon S3, to keep Git repositories small. The core improvement with Xet is its ability to deduplicate data at the byte level, significantly reducing upload and download times by only transferring modified chunks, unlike Git LFS which re-uploads entire files. The document also highlights Xet's backward compatibility with Git LFS, its security model, and usage recommendations for optimal performance, while noting current limitations like 64-bit system requirements and partial web support.
What this episode covers
The provided text discusses Hugging Face's storage solutions for large, binary files, specifically focusing on the transition from Git LFS to their new Xet-backed storage system. It explains how repositories handle massive files like models and datasets through pointer files that reference data stored remotely, typically on Amazon S3, to keep Git repositories small. The core improvement with Xet is its ability to deduplicate data at the byte level, significantly reducing upload and download times by only transferring modified chunks, unlike Git LFS which re-uploads entire files. The document also highlights Xet's backward compatibility with Git LFS, its security model, and usage recommendations for optimal performance, while noting current limitations like 64-bit system requirements and partial web support.
NOW PLAYING
Hugging Face Hub Storage: Xet vs. Git LFS
No transcript for this episode yet
Similar Episodes
Mar 14, 2026 ·23m
Mar 11, 2026 ·16m
Feb 28, 2026 ·14m