Upload model to hub
I am using the following function to upload my custom CLIP model. The issue is that given that my vision and text models have different configs, the config of the last overwrites the first. It is not taking into account thevariantargument to tag it.Any tips on the “correct” way to save this model and upload it (fingers crossed without using the git functionality).Side question: It’s also uploading a readme.md file to my understanding. Can I avoid doing this?def _upload_model_to_hub(
    vision_encoder: models.TinyCLIPVisionEncoder, text_encoder: models.TinyCLIPTextEncoder
):
    vision_encoder.save_pretrained(
        str(config.MODEL_PATH),
        variant="vision_encoder",
        safe_serialization=True,
        push_to_hub=True,
        repo_id="debug-clip-model",
    )
    text_encoder.save_pretrained(
        str(config.MODEL_PATH),
        variant="text_encoder",
        safe_serialization=True,
        push_to_hub=True,
        repo_id="debug-clip-model",
    )If it helps this is part of thesave_pretrained(intransformers/modelling_utls.py) code that is causing the issue imo. As can be seen there is no option attach the variant argument to the config.# Save the config
        if is_main_process:
            if not _REDACTED_TOKEN_config_loaded:
                model_to_save.config.save_pretrained(save_directory)
2024-05-04T12:22:58Z
To upload my models I used the following code instead. The trick was to create separate subfolders for vision and text, and then usehuggingface_hub’supload_folder(uploading the common parent) method.def _upload_model_to_hub(
    vision_encoder: models.TinyCLIPVisionEncoder,
    text_encoder: models.TinyCLIPTextEncoder,
    debug: bool = False,
):
    vision_encoder.save_pretrained(
        str(config.VISION_MODEL_PATH),
        safe_serialization=True,
    )
    text_encoder.save_pretrained(
        str(config.TEXT_MODEL_PATH),
        safe_serialization=True,
    )

    api = HfApi()
    if debug:
        repo_components = config.REPO_ID.split("/", maxsplit=1)
        repo_components[1] = f"debug-{repo_components[1]}"
        repo_id = "/".join(repo_components)
    else:
        repo_id = config.REPO_ID
    common_REDACTED_TOKEN_params = {
        "repo_id": repo_id,
        "repo_type": "model",
    }
    if not api.repo_exists(**common_REDACTED_TOKEN_params):
        logger.info(f"Creating repo {repo_id} on Hugging Face Hub.")
        api.create_repo(**common_REDACTED_TOKEN_params)  # type: ignore
    logger.info(f"Uploading models in {str(config.MODEL_PATH)} to {repo_id}.")
    api.upload_folder(
        folder_path=config.MODEL_PATH,
        **common_REDACTED_TOKEN_params,  # type: ignore
    )  # type: ignore
2024-05-05T00:23:32Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.