RoBERTa large: HF vs. FAIRseq
Hi,I’m trying to understand what the difference is between the modelhuggingface.coFacebookAI/roberta-large · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.…and the version available for download here:github.comfacebookresearch/fairseq/blob/main/fairseq/models/roberta/model.py#L39logger = logging.getLogger(__name__)@register_model("roberta")class RobertaModel(FairseqEncoderModel):@classmethoddef hub_models(cls):return {"roberta.base": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.base.tar.gz","roberta.large": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.large.tar.gz","roberta.large.mnli": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.large.mnli.tar.gz","roberta.large.wsc": "http://dl.fbaipublicfiles.com/fairseq/models/roberta.large.wsc.tar.gz",}def __init__(self, args, encoder):super().__init__(encoder)self.args = args# We follow BERT's random weight initializationself.apply(init_bert_params)For example, the HF version is 1.42GB as safe tensors, but the FAIRseq version is only 678 MB.Happy to read documentation if this is described somewhere.Thanks in advance,Dan
2024-05-09T09:35:02Z
Hi,That’s a good question. I would recommend inspecting the conversion script:transformers/src/transformers/models/roberta/convert_roberta_original_pytorch_checkpoint_to_pytorch.py at main · huggingface/transformers · GitHub.