license:cc-by-nc-4.0 arxiv:2309.08730 region:us
---
license: cc-by-nc-4.0
---

# Music Instruct (MI) Dataset

This is the dataset used to train and evaluate the MusiLingo model.
This dataset contains Q&A pairs related
to individual musical compositions, specifically
tailored for open-ended music queries. It originates
from the music-caption pairs in the MusicCaps
dataset.
The MI dataset was created through prompt engineering and applying few-shot learning techniques
to GPT-4. More details on dataset generation can be found in our paper *[MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
](https://arxiv.org/abs/2309.08730)*. 

The resulting MI dataset consists of two versions:
v1 (short questions), with 27,540 Q&A pairs seeking comprehensive
details about musical snippets including but not limited to emotion, instrument, vocal track, tempo, and genre etc., often yielding concise one or two-sentence responses. In contrast, v2
comprises 32,953 Q&A pairs featuring more general questions about the musical pieces (long questions), resulting
in typically more extensive responses that serve as
paraphrased renditions of the original caption.

## Evaluation and dataset SPlittion
You can use all (or the long/short partition of) the Q\&A pairs of which audio is in the training split of AudioSet as MI training set and use the short QA and long QA with audio in evaluation split of AudioSet as two testingsets separately. 

```
# training set
ds_mixed_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='all')
ds_long_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='long')
ds_short_train = MIDataset(processor, '/content/drive/MyDrive/music_data', split='train', question_type='short')
# testing set for short QA
ds_short = MIDataset(processor, '/content/drive/MyDrive/music_data', split='test', question_type='short')
# testing set for long QA
ds_long = MIDataset(processor, '/content/drive/MyDrive/music_data', split='test', question_type='long')
```

And the evaluation includes BLEU, METEOR, ROUGE, and Bert-Score.

## Citation
```
@article{deng2023musilingo,
  title={MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response},
  author={Deng, Zihao and Ma, Yinghao and Liu, Yudong and Guo, Rongchen and Zhang, Ge and Chen, Wenhu and Huang, Wenhao and Benetos, Emmanouil},
  journal={arXiv preprint arXiv:2309.08730},
  year={2023}
}
```
