Two way translation Speech to Speech model EN-DE
HiI am doing a project with the goal to create a model which can translate Speech to Speech in real time EN-DE and DE-EN.I have found the facebook one way translationfacebook/textless_sm_cs_en · Hugging FaceThe problem is that i tried the other way withfacebook/s2t-wav2vec2-large-en-de · Hugging Facebut it seems to be crashing.I was thinking of using a model to convert EN Speech to Text then translate the EN Text to DE Text and then a Text to Speech to output the DE Speech.I am not sure how to continue from here.Can you give me some tips?Thank you and regards
2023-09-26T11:35:52Z
I’m working at this task before but translate from English to Arabic, Build cascaded pipeline consists ofSpeech recognition for En useWav2Vecmodel.Punctuations restoration becauseWav2Vecdismiss punctuations, we usedeepmultilingualpunctuationmodel.Machine Translation from En to Ar usembartmodel.Tashkeel restoration: you can learn morehereText2Speech usingfastSpeech2model.This pipeline very slow, computationally expensive, and the result not good at all. soon i will publish this work on GitHub for further discussion. At this time we try to simplify this pipeline using SpeechT5 for End2End Speech Translation or replace the first 3 models with one model can translate En Audio to Ar text and also work on first open source Automatic video dubbing from En to Ar and vis versa at first then add support for other language.