Build a language detector
Please read the topic categorydescriptionto understand what this is all aboutDescriptionFor many online applications it is not known in advance what language an end-user will communicate in. The goal of this project is to build a system that can automatically predict the language a text is written in.Model(s)There are a few popular multilingual models that you can start with:xlm-roberta-basebert-base-multilingual-uncasedDatasetsThere are quite a fewmultilingual datasetsavailable on the Hub. Many of these have a “language” field that could be used as a target for the model to predict.ChallengesThis project will likely require you to combine several datasets together to gain enough coverage of many languages.Desired project outcomesCreate a Streamlit or Gradio app onSpacesthat can predict the language of a piece of text provided by an end-userDon’t forget to push all your models and datasets to the Hub so others can build on them!Additional resourcesA good baseline to compare your model against is the PythonlangidlibraryDiscord channelTo chat and organise with other people interested in this project, head over to ourDiscordand:Follow the instructions on the#join-coursechannelJoin the#language-detectionchannelJust make sure you comment here to indicate that you’ll be contributing to this project
2021-11-16T05:27:28Z
@lewtunHi, I would like to work on this project.
2021-11-16T09:38:40Z
Cool to hear@ivanlau! I’ve created a Discord channel (see topic description) in case you and others want to use it
2021-11-17T22:00:37Z
@lewtunHi, I’m interested in working on this project.Are we supposed to consider all the languages recognized by thelangidlibrary (which we’ll be using as baseline), or is it ok to consider fewer languages ?
2021-11-17T22:02:02Z
It’s totally fine for you to choose the scope for the projectI agree that working with a few languages is a great way to start!
2021-11-17T22:09:10Z
Hi@lewtun! I want to work on this project.
2021-11-17T22:09:30Z
Alright thanks !! I’ll do so
2021-11-17T22:19:24Z
Hi, I’d be glad to join this project. Am I still in time?
2021-11-17T22:21:02Z
I think you’re the 4th person, so yes there’s still time and space on the team!
2022-01-26T10:28:20Z
hi is this project still on ?
2022-01-26T10:30:12Z
Hi@hfawaz, this community event ended last November. Having said that, you’re more than welcome to use the#course:course-eventtopics as inspiration to build NLP powered applications
2022-01-26T10:30:56Z
thanks, do you have any shared open ressources that you guys found following this course ?
2022-01-26T10:44:27Z
You might find the official tutorials in thetransformerslibrary to be helpful:🤗 Transformers Notebooks