Pre-training for Wav2Vec2-XLSR via Huggingface
Hi guys! I note that the most topics are related to fine-tuning a pre-trained model. But if I have got some new unlabeled data, how can I preform the pre-training process via Huggingface?
2021-10-11T22:48:40Z
Hey Javen,We’ve now an official wav2vec2-pretraining example here:transformers/examples/pytorch/speech-pretraining at master · huggingface/transformers · GitHub
2021-10-13T15:48:12Z
Hi@patrickvonplaten, I re-run this scripts onGoogle colab. I pass all parameter as same as you recommend onREADMEbut after some epoch, the loss is not decrease.| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 5.137e-01| ppl: 2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 8.068e-19
| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.952e-01| ppl: 2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 4.017e-19
| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.831e-01| ppl: 2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 5.166e-19Can you take a look? what did I miss?
2021-10-17T15:27:18Z
Hi@patrickvonplatenand@tiena2cva,Thanks for the new official wav2vec2-pretraining example, this helps a lot!I had the same problem as@tiena2cva. Tried to re-run the demo script with the same parameters on my own gpu. After a few epochs the contrastive loss was decreased to zero and the model stopped changing.Running inference showed that the quantizer maps all the time steps to the same vector (can be seen at projected_quantized_states), which explains the zero contrastive loss.I would have thought that the diversity loss weight should be increased, but I used the parameters given in the README file so this behavior is unexpected and may indicate a different problem.Can you please help?
2021-11-13T17:57:53Z
Hi@patrickvonplatenand@sgugger, I am facing the same issue, constrast_loss is not changing after 3 backward it goes straight to zero and didn’t change even after many epochs, any idea how can we reproduce the results. Thanks
2021-11-13T17:58:55Z
Hi,@tiena2cvaand@ayanasany update or solution to your problem?
2022-11-14T13:13:05Z
hey@tiena2cva. Were you able to pre train your model? If yes, can you share your code?
2022-11-14T16:22:43Z
Hi@Kshitizkhandelwal, it was a long time ago, let me check in the next day. I have done pre-training wav2vec2 for Vietnamese
2022-11-15T13:41:48Z
@tiena2cvaIf possible please let me know on this?
2022-11-18T07:44:47Z
Good day! I am consnantly getting an error “realloc of size … failed” and I wanted to know is there any solution to overpass it?
2023-01-09T08:55:15Z
I have the same situation here.I tried to pretrain wave2vec on 8K and 16K samples. after few steps contrastive loss goes 0, diversity loss shoots up to 1 and perplexity to 2.I also tried changing hyper params like learning rate and gumble temp, but no luckany updates ?@tiena2cva@patrickvonplaten
2023-03-28T07:04:10Z
Do you have any updates? I’m experiencing a similar issue where the diversity loss starts off very high, around 1500, but after the warmup period, it quickly drops down to 0.0 within 10K steps. Conversely, the diversity loss starts at around 300 and gradually increases to 400, then oscillates.
2023-06-06T15:50:03Z
Hi@mohammadtvkI am experiencing the same problem did you find a solution?
2023-06-06T18:37:37Z
unfortunately no. I used fairseq for pretraining
2023-06-09T13:05:31Z
I also switched to fairseq, there are several LR and threshold management things missing in HF I think.