Is Facebook NLLB too slow?
So, I needed to have an English Translated version of some texts. For this, first, I used google trans, and this worked fine. It was able to complete about 7-8 translations per second. Then, I tried using Facebook 600M distilled NLLB. But, this took about 10 seconds for a translation. Now, I ran both the code on Google Colab. Is this the expected thing? Or is something wrong?from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline

tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")

model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")translator_nllb = pipeline('translation', model=model, tokenizer=tokenizer, src_lang="ory_Orya", tgt_lang='eng_Latn', max_length = 400)translated_text_nllb = translator_nllb(text)[0]['translation_text']
2023-08-18T17:13:12Z
I’m seeing issues with how slow this is as well. Is this expected for a 600M parameter model? I get much faster responses from other autoregressive models of similar weights. Can someone from HF chime in?
2024-01-13T11:21:06Z
I have the same issue. I wonder if this could be solved somehow.
2024-01-13T14:36:51Z
Hi,You’re not using a GPU, which will indeed make it pretty slow. Try adding thedeviceargument when instantiating the pipeline:from transformers import pipeline
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

pipe = pipeline(model="facebook/nllb-200-distilled-600M", src_lang="eng_Latn", tgt_lang='nld_Latn', max_length = 400, device=device)

translated_text = pipe("hello world")[0]['translation_text']
print(translated_text)
2024-01-14T14:49:27Z
In my case, I do have a GPU and pass it just like in your example, but it’s still too slow.
2024-01-14T17:37:10Z
Then I recommend to useCTranslate2, which is a C++ port of these models. Inference with NLLB is shownhere.
2024-04-09T11:28:29Z
nielsr:Then I recommend to useCTranslate2, which is a C++ port of these models. Inference with NLLB is shownhere.thanks, it 5 times faster.dont forget to set the devicetranslator = ctranslate2.Translator("nllb-200-distilled-600M", device='cuda')
tokenizer = AutoTokenizer.from_pretrained('facebook/nllb-200-distilled-600M', src_lang=src_lang, device='cuda')
2024-08-25T17:08:04Z
ser, have any recommend for my specs to run any model.So like this my specsAMD EPYC 7282
core : 4 vCPU
Memory space: 100GB
Memory RAM : 6GBwhat is the best run any model?
2024-08-30T08:18:53Z
Why am I performing the operation ‘ct2 transformers converter - model nllb-200-distilled-600M - output dir nllb-200-distilled-600M-CTransllate2’Reported error: AttributeError: ‘M2M100Encoder’ object has no attribute ‘embed_stale’