Ways to detect language of the given text?
Is there any way to detect the language of the given input text?There are many models to translate form one language to other.
2020-12-17T10:26:39Z
It should be fairly easy to train a small model to detect languages, not sure if there is one already but seems a bit overkill.I have used this library in the past to detect language:https://pypi.org/project/langdetect, it should be more than enough as a pre-processing step to then choose which translation model to use (if that is the use case you were referring to)
2020-12-18T13:31:45Z
Yeah i saw the package which you mentioned but I am looking for something with thetransformerbased model.
2020-12-20T16:45:53Z
Well I am not aware of any pretrained model that does what you ask (there is one toidentify programming languages). You could train your own model to do so, something like multilingualdistilbert, but again, seems a bit overkill.
2021-06-19T03:34:39Z
There are already a number of ways to get to solve this problem. We can use standard libraries. I have got a solution if you still want help in using non-transformer solutions you may get in touch with me.I am new I do not know if we can take names of other libraries in this forum.The transformer library is better used for other stuff but if we get a solution to this issue inside the transformer itself then we do not need other libraries, which is obviously of advantageous.Hope this helps.
2021-06-19T03:35:45Z
You can use textblob and fasttext libraries. Saw someone mentioning other libraries so just posting.