How to sync to the latest version with `snapshot_download`, old files removed
By default,snapshot_download(repo_id=rid, resume_download=True)download the whole snapshot of a repo’s file at itslatestversion.I was wondering, if there are new commits after its first execution, how to sync the downloaded files to the latest status and the obsoleted files are removed (to save disk space).It seems, executing the function again will sync the files to latest status (add new files toblobs/<commit-id>and update the reference links insnapshot/<commit-id>to the new files), however, the old files remains (the obsoleted files inblobskeeps).Failed to find a possible para insnapshot_download’s doc.
2023-07-20T07:52:31Z
That’s a great point! Under the hood,huggingface_hubutilizes Git for data fetching. Are you attempting to run agit clean -dfcommand to remove old untracked files? cc@Wauplin, who maintains the library, in case they can provide additional insight.
2023-07-20T08:47:46Z
Thanks for the ping@radamesActually no,huggingface_hubdoesn’t use git under the hood, except when you use the legacy classRepositorywhich is not the case here. The main download methodsREDACTED_TOKEN_download(single file) andsnapshot_download(entire repo) are HTTP-based.Here is a short explanationon the difference between the git-based and http-based approach.Regarding the initial question about how the cache work,here is a guidethat explains the cache directory structure, how it gets updated, how to scan what’s inside and finally, how toclean outdated files. I think it’s a great read to understand caching in HF ecosystem. Of course it uses the git hash/revision but it is not related or compatible with git commands.however, the old files remains (the obsoleted files inblobskeeps).This is intentional, in case users want to have different versions of a model in their cache. We believe that it’s better to let the users clean their cache themselves rather than deleting a blob file, potentially leading to a re-download.Hope this will help you
2023-07-20T09:35:08Z
Thank you for the info.@radames,
2023-07-20T12:29:02Z
@WauplinThank you for the update.Been though the points you mentioned.Considering one use python (w/ogit) for the whole process, from preparing the datasets to deploy their project, providing an optional para to the function would easy some needs on managing   dataset storage. For instance, whether or not to delete obsoleted files inblobsto reduce disk usage.It’s likely that one knows it leads to the deletions when specifying this optional para in the function.Currently, to my understanding, one would need to do it manually (cross-check the links insnapshotsand the files inblobs) or write an (bash) script to do that.
2023-07-21T13:37:36Z
Currently, to my understanding, one would need to do it manually (cross-check the links insnapshotsand the files inblobs) or write an (bash) script to do that.Not exactly. When scanning the cache, there is adelete_revisionsmethod to clean the cache given a list of revisions you want to remove. If you want to delete all files except the last modified one for each repo, you can do it like this:from huggingface_hub import scan_cache_dir


def delete_old_files():
    # Scan cache
    scan = scan_cache_dir()

    # Select revisions to delete
    to_delete = []
    for repo in scan.repos:
        latest_revision = max(repo.revisions, key=lambda x: x.last_modified)
        to_delete.extend([revision.commit_hash for revision in repo.revisions if revision != latest_revision])
    strategy = scan.delete_revisions(*to_delete)

    # Delete them
    print(f"Will delete {len(to_delete)} old revisions and save {strategy.expected_freed_size_str}")
    strategy.execute()All the logic lies into the# Select revisions to deletesection. Here I chose to delete all revisions except the last modified one. But one could also argue the way to go would be to delete all revisions except the last revision from themainbranch.The reason why I’m reluctant to implement a new parameter insnapshot_downloadis that we would have to define its exact expected behavior and that might change depending on the user. For example, if I download a model for revision tag2.0, does it mean that I want to delete1.0revision tag? Or that I want to delete revisions that are not referenced by a specific ref (main, 1.0, 2.0, another_branch,…)? Or another example, if a user downloads a previous revision of the repo, should we delete the last revision from themainbranch or not? Seems like corner cases but questions we need to answer.To avoid debates and unexpected behaviors, I would prefer to enhance thedelete-cacheCLI anddelete_revisionshelper rather than mixing it with the download methods. An issue has been created for it but TBH it has not being prioritized yet. We might reevaluate this in the future
2023-09-23T03:21:11Z
Thank you for the explanation.