How to perform fast batch inference for NLLB Model translation?
Batch Inference of NLLB Models with different source languages.Need help in inferencing NLLB models for batch inference where the source language can change. Pipeline inference is slow even on GPU. 10k examples of various languages (simple example inference) - 6 hours, batch inference of 4.5k examples of one language took 1.6 hrs.Single GPU - Tesla T4
2023-01-30T17:12:55Z
You can have a look atCTranslate2which is a Python library for fast inference. It can convert NLLB models from Transformers. See an examplehere.Disclaimer: I’m the author of CTranslate2.
2023-02-05T16:42:54Z
@guillaumeklnThanks for the reply. Awesome repo, I looked around and found it useful in understanding translation steps. But can you share an example for translation of 50/100 examples in one go and I can time it as well. I changed your given example in the repo for a batching perspective but couldn’t figure out that problem.
2023-02-06T08:11:15Z
Maybe this other tutorial is more helpful:OpenNMT – 26 Nov 22NLLB-200 with CTranslate2Tutorialsctranslate2NLLB-200 refers to a range of open-source pre-trained machine translation models. They can be used via FairSeq or Hugging Face Transformers. Recently, CTranslate2 has introduced inference support for some Transformers models, including NLLB. This...Reading time: 3 mins 🕑Likes: 7 ❤
2024-09-03T07:25:20Z
Do you have any new ideas regarding this issue now