Tiny mBART doc/info
I could not find any documentation/info for thesshleifer/tiny-mbartmodel. How big it is? How it was trained? What is the peformance, etc.? Did I miss something?
2020-07-21T15:34:48Z
AFAIK, i think this model is created for testing purpose. Pinging@sshleiferfor confirmation.
2020-07-21T15:36:27Z
…any chance for a production ready smaller mBART? Likembart-base?
2020-07-21T15:41:19Z
not that I know of.
2020-07-21T16:25:50Z
How small do you want@martin-avrios?For english-romanian or cc25?I will distill anything en-ro if you can figure out TPUtiny-mBARTis just for testing purposes, it’s randomly initialized.
2020-07-21T16:40:12Z
For cc25. I am working with mostly german and some french and italian docs and if I don’t freeze embeds thanmbart-large-cc25does not fit into the 16GB V100 (which is the largest Google Cloud has for a single GPU). Not even with batch size 1. So I thought I try a smaller mBART. But yes, TPU would be even more awesome because I could experiment with a lot more then.
2020-07-28T19:19:04Z
Yeah I have been running everything with--freeze_embeds.I’m also trying to figure out how to trim the embeddings, as most of them aren’t used, but blocked ongithub.com/google/sentencepieceHow to create new model file with restricted vocabulary?opened04:28PM - 23 Jul 20 UTCsshleiferhelp wantedfeature requestSimilar to #474, I want to restrict my vocabulary, and then save a new model fil…e that uses the restricted vocabulary.


I tried to do this by saving a vocabulary, modifying it, and then figuring out how to save the restricted model, but I found that even without any modification, running `spm_export_vocab` follow by `spm_encode --vocabulary` produces different results.

For example,
```bash
echo "Șeful ONU declară că nu există soluții militare în Siria" | spm_encode --model enro_trimmed/sentence.bpe.model
= > ▁Ș e ful ▁ONU ▁de cla ră ▁că ▁nu ▁există ▁solu ții ▁militare ▁în ▁Siria
```

```bash
spm_export_vocab --model enro_trimmed/sentence.bpe.model  --output=sp_vocab.txt
 echo "Șeful ONU declară că nu există soluții militare în Siria" | spm_encode --model enro_trimmed/sentence.bpe.model --vocabulary sp_vocab.txt
=> ▁ Ș e f u l ▁ O N U ▁ d e c l a r ă ▁ c ă ▁ n u ▁ e x i s t ă ▁ s o l u ț i i ▁ m i l i t a r e ▁ î n ▁ S i r i a
```
Is this expected behavior?
My end goal is that in python, `spm.encode_as_ids` only produces ids < length of the restricted vocab, so if there is a more direct way to achieve that objective I would love to know  it!

Thanks!I’m also happy to finetune mbart-cc25 on a public dataset for you, if that would help.Also, have you tried using Marian?
2020-07-30T06:53:20Z
Have not tried Marian yet but it seems interesting. It’s for translation, right? But since I used translation mode for my problem it could definitely work.Also I found excellent pre-trained models on TF Hub but they are not fine-tunable (according to the page). TransformerXL pre-trained on Wiki40B (a new dataset in 40 languages), separate model for each language. At least for me this would be the ultimate model. Seq2seq, unlimited sequence length and 41 languages. Seehttps://tfhub.dev/google/collections/wiki40b-lm/1
2020-08-03T22:47:03Z
yes its for translation.Try it out! It’s much smaller/faster than bart and we have 1100 language pairs:https://huggingface.co/models?search=Helsinki-NLP
2020-08-03T22:47:35Z
I’ve never actually finetuned it so let me know if there are any bugs!
2020-08-04T15:25:27Z
I noticed there is a de-de pair which is exactly what I need but I wonder who else needs this and what for? Looks like somebody already tried summarization in german?
2020-08-06T20:03:01Z
link to model?
2020-08-06T20:30:15Z
https://huggingface.co/Helsinki-NLP/opus-mt-de-de
2020-08-07T14:49:22Z
I believe that is an accident, not a summarization model.
2020-08-07T21:01:34Z
According to Jörg Tiedemann, the author, it’s a paraphraser, rather than a summarizer. He writes,There are texts with alternative translations into the same language, which I used for training intralingual models like this one. They are maybe not very useful at this moment as they probably just copy the input text. That this only obtains 40 BLEU is not that strange as this is tested with paraphrased sentences. Note also that this model is really rather a paraphrase model than a summarisation model as they seem to use it in the discussion