

TITLE
BART-fusion

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### Model description

BART- fusion, a novel model for generating lyric interpretations from lyrics and music audio that combines a large-scale pre-trained language model with an audio encoder. It uses a cross-modal attention module to incorporate the audio representation into the lyrics representation to help the pre-trained language model understand the song from an audio perspective, while preserving the language model’s original generative performance.  Please see the paper here: https://arxiv.org/abs/2208.11671 

### Open source status

- [X] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Here is the code repository for the paper: https://github.com/ldzhangyx/BART-fusion/tree/main.  The weights should be available in the checkpoints: https://drive.google.com/drive/folders/18EUUx-KT9xGJ1uq2UoOgj0X9BpngNn_T