PreTrain BART on The Pile
Description: We will pretrain a large BART model on The Pile, and measure a performance increase downstream. Potentially we could also add rotary embeddings?Model: BART (1b+)Dataset: The PileTraining scripts: Training scripts will be written as part of the project. Data processing scripts can be taken from GPT-J 6bExpected result: An adaptable JAX pipeline for training seq2seq models like BART on the pile.
2021-06-25T13:00:54Z
Im Interested in this project, Any reasons why you’d aim for BART and not any model with some architectural improvements?
2021-06-25T17:22:36Z
Great project! I’d also be interested in why one should use BART. Maybe T5 is also an option? We’ll have an official pretraining script merged for T5 very soon → see:[Flax] Add T5 pretraining script by patrickvonplaten · Pull Request #12355 · huggingface/transformers · GitHub
2021-06-25T18:14:50Z
To be honest it’s just because I am familiar with BARTI haven’t really used T5 in practice yet. A lot of a stack I built last year is still using BART.
2021-06-26T07:11:38Z
Cool idea!IMO one important consideration would be the oupt sequence length. AFAIU for BART’s denoiseing objective the output sequence length is same as the input length,where as for T5 it’s quite small, which would lead to faster training.adding rotary embeddings also seems like a good idea!Also what do you think aboutdeep encoder-shallow decoder, looking at the ByT5 paper it seems it’s worth exploring.
2021-06-26T10:48:56Z
The BART model’s objective seems to help better than T5 though, based on the results for text generation
2021-06-28T17:19:24Z
Let’s officially define this projectPutting everybody in the official sheethere. More people can still join! Leave a comment here or on the sheet if you want to change something.
2021-06-28T17:24:39Z
@THEODOROSwas interested in doing cross project collaboration, he mentioned wanting to use the BART+rotary implementation we end up with.
2021-06-29T10:40:23Z
This is a cool idea, would love to help out if I can! Adding myself to the google sheet (if thats cool)
2021-06-30T03:27:18Z
Join on the discord@morgan
2021-06-30T07:35:38Z
@valhalla@patrickvonplatenWe wanted to make some architectural improvements in the BART model, some of it related to using deberta’s tokenizer and also adding rotary embeddings, how would we use this model in HF after that?
2021-06-30T09:05:36Z
Hi@pawsFeel free to make any improvements you want. You could see howFlaxBartis implemented and try to keep the same API  (i.e__call__,save_pretrainedandfrom_pretrained), that way the model will be compatible with HF API. And for now,  you could create new repo, we could always modify the code later to make it compatible with HF API.
2021-06-30T10:29:44Z
Thanks@LouisCastricato, where is the discord link? Or do you mean the HF slack? Haven’t received an invite for that yet…
2021-06-30T15:16:06Z
You should talk to someone working at HF to get the invite. I do not think I am allowed to send or post the link.@valhallado you mind helping morgan?
2021-06-30T16:09:03Z
@LouisCastricatoHi, I have experience with using BART  and T5 for summarization before so I’m interested in this project, hope I can join.
2021-06-30T16:38:20Z
Of course! Join the discord as well!
2021-06-30T16:57:41Z
In, thanks!
2021-07-01T10:17:00Z
Added you as well@mattbui
2021-07-01T19:34:06Z
I have some experience with using BART for summarization on financial text. Very interested in learning more about pre-training seq2seq models + using TPU vms. I would love to join the project if there is still room on the team.
2021-07-01T23:50:24Z
added you