How to use the inference api on tts model?
Hi, how can I use the inference api on this model: espnet/kan-bayashi_ljspeech_vits?It receives text and should return audio file, since it is text to speech model.
2021-12-01T08:48:41Z
Hey there. Here is a snippet to do this. It will return the raw data, so you will still need to use something such asffmpegto process it. cc@Narsilimport json
import requests

headers = {"Authorization": f"Bearer {API_TOKEN}"}
API_URL = "https://api-inference.huggingface.co/models/espnet/kan-bayashi_ljspeech_vits"

def query(payload):
    data = json.dumps(payload)
    response = requests.request("POST", API_URL, headers=headers, data=data)
    return response.content

data = query({"inputs": "test"})
2021-12-01T15:43:37Z
@osansevieroHey! Thanks for the help. I managed to call the api and to get a response:)Regarding processing of the raw data, does it mean I should run it throw ffmpeg and create an audio file from it, or you meant something different? Thanks again!
2021-12-01T15:57:31Z
@Narsilmight know what’s the most recommended way to load the audio in this point. I usually useffmpegbut in the context of small tests and quick debugging. I’ll let Nicolas give a more proper answer.Btw, we have an audio study group forming in our Discord server:Hugging Face. The first session will be in two weeks in case you’re interested.
2021-12-03T09:54:18Z
@danijelpetkovic,At this point you should have received a real audio file. If you just want to listen you can just save itwith open("out.flac", "wb") as f:
    f.write(response.content)And in order to listen to it, use your favorite media player.vlc out.flacfor me. Windows media player, winamp, foobar, chrome and firefox can probably read it too.ffmpegis necessary only if want multiple files and want to compress them a little. The API responds flac files, which are lossless compression, but you can get much better compression with lossy algorithm (all to save disk, it might not be necessary to you)
2021-12-14T18:36:16Z
@NarsilThank you very much! I managed to save and play the file exactly as you described above. The only thing that I still can’t figure out is why the audio is distorted, it is not the same voice as it was when I was calling the model directly. I tried to add parameters to the inference api:audio_file = query_audio_tts({
            "inputs": generated_answer,
            "parameters": {
                "vocoder_tag": "str_or_none(none)",
                "threshold": 0.5,
                "minlenratio": 0.0,
                "maxlenratio": 10.0,
                "use_att_constraint": False,
                "backward_window": 1,
                "forward_window": 3,
                "speed_control_alpha": 1.0,
                "noise_scale": 0.333,
                "noise_scale_dur": 0.333
            }
        })But this didn’t help. Do you have any ideas here? Thanks!
2021-12-14T20:07:06Z
@danijelpetkovicDo you have a reference for how you create your files without distortion ?Currentlyespnetand other libraries (except transformers) don’t support adding parameters unfortunately. Since every library will have a different set of parameters, and maintaining each and everyone of them will very fast become tedious and probably clashes and bugs (not even mentioning the docs for that).That being said, if we can improve the defaults for some models we should definitely do so, the implementations for this API is defined here:huggingface_hub/automatic_speech_recognition.py at main · huggingface/huggingface_hub · GitHubAs you can see it’s pretty bare bones, any help to improve that is welcome.
2021-12-15T09:41:41Z
@NarsilYes, sure. Here is the code chunk which I was using before:from espnet2.bin.tts_inference import Text2Speech
from espnet2.utils.types import str_or_none
import torch
import scipy.io.wavfile

tag = 'kan-bayashi/ljspeech_vits' 
vocoder_tag = "none"

text2speech = Text2Speech.from_pretrained(
    model_tag=str_or_none(tag),
    vocoder_tag=str_or_none(vocoder_tag),
    device="cpu",
    # Only for Tacotron 2 & Transformer
    threshold=0.5,
    # Only for Tacotron 2
    minlenratio=0.0,
    maxlenratio=10.0,
    use_att_constraint=False,
    backward_window=1,
    forward_window=3,
    # Only for FastSpeech & FastSpeech2 & VITS
    speed_control_alpha=1.0,
    # Only for VITS
    noise_scale=0.333,
    noise_scale_dur=0.333,
)

def get_audio_tts(text):
  with torch.no_grad():
    wav = text2speech(text)["wav"]
    scipy.io.wavfile.write("out.wav",text2speech.fs , wav.view(-1).cpu().numpy())
  return  "out.wav"

audio_file = get_audio_tts(generated_answer)
2021-12-15T09:58:41Z
Those options do seem to depend on the model looking at the comments.Do you think you could maybe create a PR here:huggingface_hub/automatic_speech_recognition.py at main · huggingface/huggingface_hub · GitHub?We could ping someespnetmaintainers to take a look.
2021-12-15T15:47:31Z
@NarsilI created a new issue and referenced the file you sent me.github.com/huggingface/huggingface_hubUse espnet/kan-bayashi_ljspeech_vits model via inference apiopened03:46PM - 15 Dec 21 UTCdanijelpetkovichttps://github.com/huggingface/huggingface_hub/blob/eeeb0d1b352fb3249541bb62f1f5…7f41ae3ab4e0/api-inference-community/docker_images/espnet/app/pipelines/automatic_speech_recognition.py#L8

Hey guys. I tried to use the mentioned model via inference api and the audio that comes as an response is distorted.

When I try to import the model and call it directly it returns a nice audio with nice voice:

```
from espnet2.bin.tts_inference import Text2Speech
from espnet2.utils.types import str_or_none

tag = 'kan-bayashi/ljspeech_vits'
vocoder_tag = "none"

text2speech = Text2Speech.from_pretrained(
  model_tag=str_or_none(tag),
  vocoder_tag=str_or_none(vocoder_tag),
  device="cpu",
  threshold=0.5,
  minlenratio=0.0,
  maxlenratio=10.0,
  use_att_constraint=False,
  backward_window=1,
  forward_window=3,
  speed_control_alpha=1.0,
  noise_scale=0.333,
  noise_scale_dur=0.333,
)

```
2021-12-16T09:47:37Z
Turns out the issue lied with the sampling rate in fact. Here is the fixFixing FS for `espnet`. by Narsil · Pull Request #542 · huggingface/huggingface_hub · GitHub
2021-12-16T10:03:08Z
@NarsilThanks for a fast response and the fix. I will test it in the app and let you know!
2021-12-20T13:01:06Z
@NarsilHey. Sorry if I am missing something, maybe I misunderstood. But I tried the response of the api after the fix from above and it still keeps returning distorted voice.
2021-12-23T17:41:13Z
@NarsilHeyI found an interesting thing and I created a small app to show the problem. Depending on the content sent to the tts model, the voice is being returned differently.huggingface.coTest Tts Inference Api - a Hugging Face Space by danijelpetkovicDiscover amazing ML apps made by the communityYou can just try to paste this 2 slightly different paragraphs and you will see the difference in the voice.“Water heated to room temperature feels colder than the air around it. This is because the temperature difference between the water and the air is greater than that of the air surrounding it.”“Water heated to room temperature feels colder than the air around it. This is because the temperature difference between water and air is greater than the difference between the temperature of the water and the air.”
2022-01-03T13:58:01Z
This looks like a cache issue (There’s a cache in front of the API to prevent calculating things over and over).You can try adding{"inputs": "....", {"parameters": {"use_cache": False}}to your input to force the output to be calculated.The caching mechanism should be upgraded at some point so you don’t have to do this.