BART with custom encoder and decoder
Hi, I am currently working on german abstractive summarisation. My goal is to have a custom style abstractive summarisation model that learned a certain style of summarisation.I was therefore thinking if it is possible to train a German or multilingual GPT-2 model for language modeling and inserting it into a BART model as the decoder and a fine tuned BERT model as encoder.BART always states to use a “GPT-like decoder” and “BERT-like encoder”, which made me wonder if the models are exchangeable. That would allow me use a GPT-2 decoder trained on the specific task and in the desired language and a German BERT model as encoder.Does anyone have any experience on doing so? Also if only one part is exchangeable (decoder or encoder) this would also be very interesting.I am happy for any leads.CheersLukas
2022-06-09T09:46:08Z
By simply calling model.decoder = gpt_model I would add the decoder to the model but not exchanging it, is that correct?
2023-05-24T15:12:11Z
It’s BARD, not Bart. Bart is a cartoon character from The Simpsons as well as Bert, a puppet character from Sesame Street.
2023-05-24T15:34:35Z
huggingface.coBARTWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2023-05-24T15:35:31Z
It is BART not Bart…
2023-05-25T20:56:41Z
Well, it also means “Bay Area Rapid Transit”. But for me it’s funny a model that has a popular cartoon character’s name. Ay Caramba!.