text
When the XetHub crew joined Hugging Face this fall, 
When the XetHub crew joined Hugging Face this fall, 
mention
@erinys
erinys
text
  and I started brainstorming how to share our work to replace Git LFS on the Hub. Uploading and downloading large models and datasets takes precious time. That’s where our chunk-based approach comes in.
  and I started brainstorming how to share our work to replace Git LFS on the Hub. Uploading and downloading large models and datasets takes precious time. That’s where our chunk-based approach comes in.
new_line
new_line
text
Instead of versioning files (like Git and Git LFS), we version variable-sized chunks of data. For the Hugging Face community, this means:
Instead of versioning files (like Git and Git LFS), we version variable-sized chunks of data. For the Hugging Face community, this means:
new_line
new_line
text
⏩ Only upload the chunks that changed.
⏩ Only upload the chunks that changed.
new_line
text
🚀 Download just the updates, not the whole file.
🚀 Download just the updates, not the whole file.
new_line
text
🧠 We store your file as deduplicated chunks
🧠 We store your file as deduplicated chunks
new_line
new_line
text
In our benchmarks, we found that using CDC to store iterative model and dataset version led to transfer speedups of ~2x, but this isn’t just a performance boost. It’s a rethinking of how we manage models and datasets on the Hub.
In our benchmarks, we found that using CDC to store iterative model and dataset version led to transfer speedups of ~2x, but this isn’t just a performance boost. It’s a rethinking of how we manage models and datasets on the Hub.
new_line
new_line
text
We're planning on our new storage backend to the Hub in early 2025 - check out our blog to dive deeper, and let us know: how could this improve your workflows?
We're planning on our new storage backend to the Hub in early 2025 - check out our blog to dive deeper, and let us know: how could this improve your workflows?
new_line
new_line
link
https://huggingface.co/blog/from-files-to-chunks
https://huggingface.co/blog/from-files-to-chunks
text
When the XetHub crew joined Hugging Face this fall, @erinys  and I started brainstorming how to share our work to replace Git LFS on the Hub. Uploading and downloading large models and datasets takes precious time. That’s where our chunk-based approach comes in.

Instead of versioning files (like Git and Git LFS), we version variable-sized chunks of data. For the Hugging Face community, this means:

⏩ Only upload the chunks that changed.
🚀 Download just the updates, not the whole file.
🧠 We store your file as deduplicated chunks

In our benchmarks, we found that using CDC to store iterative model and dataset version led to transfer speedups of ~2x, but this isn’t just a performance boost. It’s a rethinking of how we manage models and datasets on the Hub.

We're planning on our new storage backend to the Hub in early 2025 - check out our blog to dive deeper, and let us know: how could this improve your workflows?

https://huggingface.co/blog/from-files-to-chunks 
🔥
assafvayner andrewrreed garrethlee Sri-Vigneshwar-DJ port8080 victor shawon John6666 Joseph717171 julien-c davidberenstein1957 Norod78 johnlockejrr davanstrien erinys BrigitteTousi clem Dref360 ZennyKenny
❤️
assafvayner andrewrreed Joseph717171 julien-c davidberenstein1957 BrigitteTousi clem Dref360
🧠
assafvayner ArthurZ Joseph717171 davidberenstein1957 BrigitteTousi clem Dref360